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Foreword 

This is the 10th annual report to the Congress on federally funded education 
programs. These programs were transferred in May 1980 to the U.S, Department 
of Education from the U.S. Office of Education. This year's report, like its 
predecessors, responds to several congressional mandates. The original directive 
for the Annual Evaluation Report is contained in Section 417 (a) of the General 
Education Provisions Act (GEPA). The Report is to evaluate the effectiveness of 
programs in achieving their legislated purposes and include recommendations for 
achieving greater effectiveness. Section 1246 of the Education Amendments of 
1978 (Public Law 95-561) modified the original GEPA mandate to require that 
reporting on program effectiveness also Indicate compliance with provisions of 
the law on maintenance of non-Federal expenditures where such provisions exist. 
A third requirement became law early in fiscal year (FY) 1981 as part of the 
Education Amendments of 1980 (Public Law 96-374). Section 1305 further 
amends the GEPA mandate to require tabulations of available data to indicate 
program and project impact by the sex, race, and age of beneficiaries. 

In addition to meeting these mandated requirements, this report describes the 
role of evaluation in improving education programs and changes in evaluation 
activities in the new Department of Education (ED). It also explains the 
evaluation mission in ED. 

Users of the previous, single-volume editions of this report will quickly note that 
this year's version has been prepared and packaged differently. Past Office of 
Education (OE) reports were published in one volume. They. contained overviews 
of evaluation activities with detailed program material comprising the bulk of 
the volume. The FY 1980 report has been divided into two volumes. The new, 
condensed volume I is intended for general distribution. Detailed program-by- 
program summaries are now contained in a second volume which will be 
distributed to appropriate congressional committees and to other interested 
readers upon request. 

Several reasons underlie this change in format. The evaluation report is read by 
a diverse audience with differing needs and uses for evaluation information. 
Congressmen, State and local officials, the education community, parents, 
students, and concerned citizens all have a general interest in the effectiveness 
of Federal education programs. Specific information needs, however, differ 
widely and detailed reporting is critical only in some instances. The two-volume 
report will enable ED tO’ respond .to varying information requirements, depending 
upon the needs of interested parties. The Department believes that the two- 
volume report will be a more flexible, efficient, and cost-effective way to 
disseminate evaluation information than the large, single volume. 

Because volume I is intended to provide a broad overview of ED evaluation 
activities, additional explanatory material has been included. This first volume 
contains information on the Department's innovative information gathering and 
evaluative techniques, and on its management initiatives to better utilize 
evaluation findings. Highlights of new evaluation findings and the uses of ED 
evaluations can be found in chapter 3. A general perspective on major ED 
program evaluation activities is contained in chapter 2. Program areas covered 
include elementary and secondary education, postsecondary education, and 
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diverse special category programs. As previously mentioned, detailed 
information on individual program evaluations which were in past annual 
evaluation reports has been updated for FY 1980 and appears as volume II. All 
programs that were formerly the responsibility of the Office of Education are ^ 
covered in this year's report. The FY 1981 report, representing the Department s 
first full fiscal year of operation, will contain evaluation reports on these 
programs as well as comparable reports on programs transferred to the 
Department from other Federal agencies as a result of the ED Organization Act. 

I would like to acknowledge the contributions of several individuals to this par's 
Report. Dr. Thomas Uhlman translated the idea of a first volume into reality by 
structuring and writing much of it. The Evaluation Coordination Staff, in 
particular Mr. Edward Classman, Ms. 3eane Onufry, and Ms. Elaine Green, helped 
pull together and organize diverse evaluation materials from a variety of 
sources. Evaluation division directors and their staffs contributed specific 
program chapters and overview materials. Finally, Ms. Diana Carpenter's 
tireless typing was essential to the successful completion of this report. 


John Seal 

Acting Assistant Secretary 
for Management 


For copies contacts Ms. Yvonne Briscoe 

Office of Evaluation and Program Management 
Rooni ^070, Switzer Building 
U.S. Department of Education 
Washington, DC 20202 
(202) 2^5-0361 
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Chapter I 

Evaluation in the 
Department of Education 




On May 4, 1980, the Department of Education (ED) became the 13th Cabinet- 
level Department in the U.S. Government. Today, the Department's numerous 
programs focus on two primary goals. The first is to assure all students, 
regardless of age, background, or native ability, equal access to the best possible 
education. The second goal is to help improve the quality of education for every 
student. 

One critical component of accountability in Federal education activities in the 
Office of Education (OE) and now in ED is evaluation. The legislation 
establishing the Department specifically cites improvements in evaluation as a 
purpose for its creation. Broadly defined, evaluations are objective assessments 
of program and management performance. Specific approaches adopted, costs 
incurred, and study objectives vary, but all ED evaluation activities share the 
common purpose of trying to ensure that Federal education monies are being 
efficiently spent on programs that are successfully meeting their congressional 
mandates. 


U.S. OFFICE OF EDUCATION 

Education evaluation requirements were first mandated by the Congress in 1965 
under Title I of the Elementary and Secondary Education Act. This legislation 
included a provision that school district grantees evaluate the effectiveness of 
their programs. Other OE programs soon contained evaluation requirements as 
well. The FY 1970 budget was the first to contain a specific line-item for 
evaluation funds. That year OE also centralized evaluation activities in the 
newly established Office of Program Planning and Evaluation, later renamed the 
Office of Evaluation and Dissemination. This office assumed the lead in 
directing an increased number of national education evaluations. The staff 
designed studies, selected contractors to do fieldwork and analyses, monitored 
contractor performance, and reported on study findings and recommendations. 
From 1970-79, OPPE spent an average of $18.4 million a year on evaluation 
activities. 

Not all education evaluation activity took place in the Office of Evaluation and 
Dissemination or its predecessor offices. The Bureau of Education for the 
Handicapped sponsored studies on the implementation of the Education for All 
Handicapped Children Act} the Bureau of Student Financial Assistance conducted 
quality control studies of the Basic Educational Opportunity Grant program; and 
the National Institute of- Educatipn and the Follow Through program supported 
evaluation research and some program evaluations. 


U.S, DEPARTMENT OF EDUCATION . 

During the 1970's, evaluation became a critical component of federally sponsored 
education programs, and evaluation is retaining its prominent position in the 
Department of Education as the I980's unfold. ED evaluation activities are the 
responsibility of the Office of Management under the Deputy Assistant Secretary 
for Evaluation and Program Management. Several reasons prompted this 
organizational response to the new Department's evaluation mission. 
Organizational separation between the offices in charge of program operations 
and the assessment of those programs were deemed essential to maintain the 
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independence and objectivity of program reviews. Second, the piacement of 
evaluation activities within the Office of Management reflects a departmental 
commitment to translate evaluation findings into management improvements 
whenever possible. The Office of Evaluation and Program Management (OEPM) 
has undertaken a series of initiatives that bring together evaluation, policy, and 
budget staffs to explore the implications of program evaluations on all aspects of 
current departmental activities and to incorporate evaluation results into future 
planning. Finally, a number of management evaluation efforts, also highlighted 
in this report, gather information on program operations that can be used in 
conjunction with the more elaborate program evaluation studies. Placing 
management and program evaluation activities within the same office allows the 
Department to draw more effectively upon a variety of evaluation techniques to 
meet specific information needs. 

Because of their complexity and data collection requirements, impact and 
process evaluations (definitions and discussion follow) are conducted by outside 
research organizations. These studies usually assess the national impact of 
programs and/or the manner in which they have been implemented. They are 
administered by three divisions in the Office of Evaluation and Program 
Management: the Elementary and Secondary Programs Division; the 
Postsecondary Programs Division; and the Occupational, Handicapped, and 
Developmental Programs Division. 

The functions of these program evaluation divisions include: 

o Reviewing the state of knowledge about programs and identifying 
program information needs. 

o Preparing study designs and contractor selection criteria. 

o Directing the progress of contracted work with particular 

attention to the implementation of the original study design and 
technical direction to guide changes in design (this involves the 
selection and development of measuring instruments, sampling, data 
collection, and analysis). 

o Preparing appropriate executive summaries to communicate findings 

to ED policymakers, the Congress, and the public. 

These three divisions constitute the Office of Program Evaluation, one of two 
major offices reporting to the Deputy Assistant Secretary for Evaluation and 
Program Management. 

The counterpart office under the same Deputy is the Office of Management 
Evaluation. Several critical Department management functions are also housed 
within this office which includes the Divisions of Program .Assessment, Quality 
Assurance, Organizational Development, Management Analysis, and Education 
Data Control. Among their responsibilities aret^ 

o management analyses 

o quality assurance studies 

0 needs analyses for office automation 
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o work measurement reporting 

o short-term management assessments of how well programs 

are being delivered to recipients 

o organizational development planning 


ED EVALUATION ACTIVITIES 

The concept of evaluation has been broadened considerably in the Department of 
Education. New techniques and approaches have been developed to increase the 
capabilities of Department policymakers and program officials to meet program 
information needs under budget and time constraints. Before explaining the new 
techniques or the old, one must review how the Department goes about 
establishing an evaluation agenda. The evaluation planning process structures 
the agenda and largely determines the choice of evaluation techniques to be 
used. 

Evaluation Planning 

Evaluation information is necessary in order to answer one or more critical 
questions about education programs. These questions remain at the heart of the 
evaluation planning process: 

0 Who is served by the program and who needs service? 

o What are the services, how well are they delivered, and 
what do they cost? 

o What are the effects of services on recipients? 

o What are the costs and benefits of alternatives? 

o What improvements should be made in the management and 
administration of the programs? 

o What program practices, projects, or products are effective 
in achieving program objectives? 

Every education program would' profit greatly with answers to questions. 
Budgetary constraints, however, have meant that programs could only be 
evaluated on a selective basis. Because of competing program demands, 
evaluation planning began in the Office of Education. 

In OE, the strategy originally used to identify program evaluation needs was for 
the central evaluation staff in the Office of Program Evaluation to request 
annually formal recommendations for evaluation projects from program 
managers and then develop a recommended list of projects to be undertaken in 
the following year. This list was submitted to the Assistant Secretary for 
Planning and Evaluation (ASPE) in the former Department of Health, Education, 
and Welfare (HEW) for approval. 
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In January 1979, OE initiated a new procedure to develop the annual evaluation 
plan. The purpose of the new procedure was to exert more control over 
evaluation planning and broaden the review process. An Evaluation Planning 
Committee was established consisting of several OE and HEW policy officials. 
This group prepared a comprehensive plan for ASPE's approval. The expansion of 
OE's evaluation planning and review process was generally regarded as a useful 
development in that it helped focus OE planning on such critical factors as 
program reauthorization and specific congressional or departmental concerns. 

One problem that any evaluation group must solve is how to ensure the 
timeliness of their evaluation findings for decisionmakers. A common criticism 
of most Federal evaluation offices has been that their study findings lose 
relevance to Administration and congressional policy officials because results 
are reported too late to be fully used in reauthorizing programs or deciding on 
budget requests or appropriation levels. In response to this problem, the Office 
of Evaluation and Program Management has developed an evaluation planning 
model. Its primary purpose is to help ED focus on the type and timing of 
program and policy questions to be addressed by evaluation staff, so as to meet 
the information needs of executive branch and congressional policy officials. 

The approach adopted for the model is substantially more systematic and 
comprehensive than previous efforts. In addition to helping ensure that 
appropriate information from management and program evaluations is available, 
the planning model can also be used for planning, research, and statistical studies 
conducted by other ED offices. 

The context for the evaluation planning model is the "life cycle," as authorized 
in legislation, of individual Federal education programs or groups of programs 
under consideration. Life cycles begin approximately 2 years before a given 
program is to be reauthorized and continue ’‘ntil the following reauthorization. 
Accordingly, a typical life cycle is 5 to 7 years, depending upon the length of 
reauthoflzation. 

There are two reasons why the life cycle and, consequently, activities under the 
evaluation planning model start even earlier than the beginning of a 
reauthorization period. One is the importance of specifying in advance the 
information needs pertaining to future legislation, management, and budget 
issues and identifying the sources for each type of information. This 
specification must be done far enough in advance for each information-producing 
activity to be planned, carried out, analyzed, and reported in a timely fashion. 
The second reason is to foster early coordination of information-producing 
activities, thereby minimizing gaps or duplication of effort and to ensure that 
requests to the field for data are not excessive. The evaluation planning model 
also recognizes the need for flexibility during the program's life cycle as the 
information agenda changes and corresponding shifts are made in the evaluation 
plan, so that management and program reports will be timely, relevant, and of 
the greatest possible use. In addition, there are a variety of uses for evaluation 
findings aside from reauthorization efforts. For example. Department managers 
use evaluations to improve grant-making and other processing operations while 
the budget process demands information from evaluations for Department, 0MB, 
and congressional officials. As information needs change during a program's life 
cycle, revisions in the application of the evaluation planning model to particular 
program areas can be made. Overall, the planning model serves as the guide for 
selecting among various evaluation techniques to fulfill the information agenda. 
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Traditional Evaluation Activities 


Evaluation information on national programs is essential for responsible 
management. 'Without it, program effectiveness would remain unknown or 
anecdotal and improvements in service delivery would be hampered. A variety 
of information-gathering techniques or approaches are required in order to meet 
specific management needs. Some can be categorized as traditional in that they 
have been developed by evaluators over a relatively long period {15-20 years) and 
are viewed by many professionals as the "standard" way of gathering program 
information. These activities are called process or impact evaluations. 

Process evaluations include studies of the activities, operations, organization, 
and other aspects of a program. They may be designed to examine reported 
shortcomings in program operations or merely to monitor program 
implementation. Less elaborate varieties may involve short site visits and 
management case studies. The more elaborate approaches can involve careful 
measurement of how many individuals receive services or how institutions 
comply with Federal mandates. The main focus of such activity is understanding 
the delivery of program services and adherence to standards, rather than 
understanding or estimating effects of service. 

Impact evaluations of education programs attempt to estimate as accurately as 
possible the direct effects of a program on its main target group, whether 
children, adults, or institutions. These studies are more difficult when there are 
multiple target groups and when many program effects are indirect. If costs and 
effects of program variations are estimated, the activity may involve cost- 
benefit analyses. 

During FY 1980, 13 impact evaluations were in progress. The studies were 
conducted by highly qualified, professional research firms or universities 
according to explicit requirements detailed by the Department's evaluation staff. 
It may be asked why Federal employees do not carry out these projects rather 
than work through an outside contractor. The answer is that program 
information needs (both in terms of the number of programs that are evaluated 
and the complexity of each evaluation) and the consequent collection of data in 
the field far outstrip the relatively limited evaluation staff resources within the 
Department. 

Although they do not actually lead a research team in the field, departmental 
evaluation staff provide technical direction and control from the beginning to 
the conclusion of a process or impact evaluation. Initially, a staff member is 
designated as the project monitor and develops the specifications for the work to 
be performed by a contractor. Once the contractor is selected through a 
competitive bid process and the evaluation is initiated, the project monitor 
maintains close contact with the contractor throughout each phase of the study. 
The basic purposes for this monitoring function are: (1) to make sure that the 
study is conducted in accordance with the original design or approved changes; 

(2) to advise the contractor when problems arise which may jeopardize the 
validity of the study; and (3) to ensure that the study, as executed, answers the 
appropriate questions. At the completion of the contract, the monitor takes 
responsibility for the production of an executive summary of the findings, and 
ensures that the contractor's report is reviewed by Department officials and is 
ultimately transmitted to the appropriate audiences. 
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The greatest strength of impact and process evaluations is that they provide the 
most comprehensive view of program operations or accomplishments that the 
Department can obtain. On the other hand, two drawbacks may make them 
inappropriate in certain situations. Because of the elaborate nature of these 
studies, they are more expensive than other options, and cost is an even more 
critical factor today when evaluation budgets have generally been reduced. 

Time is another consideration that can work against impact and process 
evaluations. They may take several years to complete, and information needs 
are often more immediate because many program and budgetary decisions cannot 
be deferred. 

These shortcomings have led evaluation officials in the Department to search for 
other means to obtain useful evaluation information more quickly and at less 
cost. The result is a series of evaluation/management innovations that 
supplement impact and process evaluation capabilities within OEPM. 

New Evaluation and Management Activities 

Additional program information is being obtained through four evaluation 
techniques new to the Department of Education. Evaluabillty assessments and 
the program objectives system focus on developing a broad consensus regarding 
long-range, measurable program goals and objectives. Program management 
reviews and service delivery assessments are low-cost evaluation tools which 
Department staff also use to obtain program information. In addition, 
management evaluation techniques are used to review and analyze the 
effectiveness and efficiency of ED's management practices, systems, 
organization, and procedures, and to prescribe improvements where appropriate. 
These techniques include; management analyses; quality assurance studies; and 
organizational development activities . 

1. Evaluability Assessment (EA) 

EvaJuability assessment is a technique designed to support the evaluation process 
by first ensuring that a program has a solid management foundation, EA's 
purposes are to determine the extent to which a program is ready for evaluation; 
what changes might be needed to make the program more manageable and 
accountable; and how an evaluation of the program might be most usefully 
conducted. A fully successful evaluability assessment results in: (1) clearly 
specified and agreed upon program objectives and activities; (2) an explicit 
statement of the assumptions that underlie the program; (3) a list of program 
performance indicators that are agreed upon by those responsible for the 
program; and (4) a detailed plan for 

o What are management’s program objectives and expectations? 

What resources, activities, objectives, and assumptions make up 
management's intended program? 

o What, in the view of policymakers (executive and legislative), 

IS the program expected to accomplish, what are acceptable 
indicators of performance, and how will program informa- 
tion be applied to them? 

o What are the likely uses of information on program performance 
at each management level? What range of actions might 
management consider as a result of various findings? 
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On the basis of resporvses to these questions and a review of relevant documents 
{e.g., the authorizing legislation and legislative history), the assessment team 
develops three types of descriptive charts or models of the program. A logic 
model is drafted to represent the intended logic of the program (e.g.j if event A 
occurs, then it is assumed that event B will occur). A second set of models, 
known as function models, trace the program's processes, including such events 
as flows of activities, people, money, and information. The third model deals 
with measurement of progress toward program objectives. It identifies measures 
which could be taken at various points in the process to indicate program 
performance. Since all these models are based on interviews with Federal 
officials only, they represent a description of the intended program . 

The assessment team then visits a limited sample of projects to obtain 
information about the actual program as it operates in the field. A second set of 
models is developed to represent program reality. Questions addressed during 
and after the field visits include: 

o What are the program inputs, activities, and outcomes 

based on a review of actual operations at a sample of project 
sites? 

0 What measurements and comparisons are feasible, given existing 
data systems? 

o What data are obtainable on program performance? Are there 

data sources for management's agreed-upon measures of progress? 

All information gathering and modeling at the policy, management, and field 
operations levels lay the groundwork for the analytical stage of the assessment. 
Questions asked by the team during the analysis of program evaluability include; 

o Is program management's description well-defined? 

o Is management's description acceptable to policymakers? 

o Is management's description a sound representation of the 
program in the field? 

o Are management’s expectations plausible ~ that is, do they 
appear likely to be accomplished? 

o What portion of the intended program is ready for useful 
evaluation? 

o What management options can be suggested to improve program 
performance? 

Outputs of an evaluability assessment generally include: (1) an agreed-upon 
program description including objectives, activities, and anticipated outcomes; 

(2) possible measures of program performance; and (3) suggestions for improving 
performance and measuring accompiishments- 

Like full-scale impact and process evaluations, EA's are conducted by an outside 
team of researchers subject to close supervision by a Department project 
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monitor. Evaluability assessments diifer significantly in that they are intended 
to produce results relatively quickly (6 months) and inexpensively (an average 
cost of $65,000). Their greatest benefit, however, is the savings of time and 
money that might otherwise be expended initially on an unsuccessful impact or 
process evaluation — that is, the resources that might be used to evaluate a 
program that instead is shown by an EA to lack agreed-upon, measurable 
objectives. 

Twelve EA studies of ED programs were initiated in FY 1979 and FY 1980. At of 
the end of FY 1980, work was still in process on two of the assessments (the 
Strengthening Developing Institutions program and the Women's Educational 
Equity program); two others had been terminated before completion (the 
Vocational Education program and National Center for Education Statistics);^ 
eight of the assessments had been completed. These eight studies covered the 
following programs: Bilingual Education; Career Education; Cooperative 
Education; Early Childhood Education for the Handicapped; Follow Through; 
Institute for Museum Studies (General Operating Support); Language Training and 
Area Studies; and Vocational Rehabilitation. At least four new assessments will 
be undertaken in FY 1981. 

2. The Program Objectives System 

The Office of Evaluation and Program Management is currently developing a 
system that is less intensive, time consuming, and expensive than evaluability 
assessment but, like EA's, is intended to generate measurable program objectives 
and indicators of achievement. The process will place primary responsiblity on 
program managers for: (1) identifying key objectives; (2) developing measures or 
indicators which can chart progress; and (3) preparing annual progress reports for 
departmental and congressional review. OEPM will be responsible for: installing 
and managing the system; defining terms and setting criteria for objectives; 
providing technical assistance and necessary support to program managers; 
coordinating review of objectives and indicators; and advising on the general 
quality and appropriateness of final products. 

It is hoped that this program objective-setting effort will produce objectives and 
indicators for programs just prior to ED's internal budgeting cycle so that the 
objectives are useful in the budget review and policy activities of the 
Department. In addition, the Department will be able to improve significantly 
its efforts to meet the mandate of Section 417 of the General Education 
Provisions Act requiring development of specific objectives and performance 
indicators for all ED program's. 

The objectives and indicators developed through this process will be transmitted 
to the Congress in the Annual Evaluation Report . It is anticipated that the 
transmission of these individual program objectives and measures will assist the 
Congress and the Department in agreeing upon legislative and administrative 
mandates for program operations. Some differences will naturally emerge in this 
dialogue, but the existen'^e of the initial objectives should be helpful in clarifying 
and resolving those differences. 

Programs which are especially complex or controversial will continue to have 
objectives developed through the more analytical, intensive process of 
evaluability assessment. The program objectives system should be extremely 
cost-efficient because, once developed, the entire process can be operated 
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exclusively by Department management and program staff. Through the use of 
both methods, the Department hopes to improve the management of its services 
and programs. 

3. Program Management Reviews (PMR) 

This evaluative technique is also being developed and refined by the Office of 
Evaluation and Program Management. It is a study designed to synthesize 
existing program information in a short period of time and to supplement that 
information with independent, short-term assessments covering important 
knowledge gaps about the operations of the program. The resulting report gives 
the Secretary and other top managers a complete but succinct picture of a 
program including its history, legislative goals and objectives, resources, 
administrative and management operations, field perceptions, effectiveness, 
issues, and problems. 

Program management reviews will be particularly valuable: when programs are 
facing reauthorization without sufficient data on certain aspects of their 
operations; when controversial Issues require difficult management decisions; 
and when top management changes are made. While PMR's are not designed to 
develop quick solutions for program problems, they can be used as diagnostic 
tools to identify where administrative, policy, or legislative changes should be 
made. 

A PMR will be directed by OEPM staff with the assistance of other program and 
staff offices within ED. Staff with expertise in management and program 
analysis, work measurement, organizational behavior, quality control techniques, 
and other fields make up a PMR team. 

The specific purpose, scope, level of detail, and length of each PMR will be 
defined by the Office of Evaluation and Program Management in conjunction 
with top management for the program. Because each PMR will require a 
significant level of resources, no more than three or four such studies will be 
undertaken in a fiscal year. Use of the PMR technique will be reserved for high 
priority needs and interests of the Secretary and Assistant Secretaries. 

4. Service Delivery Assessments (SDA) 

These short-term, current assessments of ED programs and program-related 
issues are conducted for the immediate use of the Secretary and Under 
Secretary. The 3- to 5-month studies collect opinions and suggestions from 
persons who directly receive or provide services under ED programs in order to 
gain an understanding of how programs are perceived across the country. An 
SDA gauges: how successful ED programs are perceived to be at the State and 
local levels in meeting the needs of students, teachers, parents, and 
administrators; what problems exist as programs are being implemented; and 
what improvements are identified by those directly affected by the Federal 
programs. 

It is Important to understand that SDA's are limited activities that do not 
correspond to traditional evaluation studies, audits, compliance reviews, or 
monitoring activities. While they often employ similar methods, SDA's are more 
analogous to in-depth analytical reporting which uses open-ended discussions 
with people in local settings. The knowledge gathered is generally subjective and 
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qualitative in nature. Because of these characteristics, SDA findings are not 
intended for general distribution but rather for use as one of many program 
information sources available to the Secretary and other decisionmakers. 

The reasons for undertaking an individual SDA vary with each study. Some 
frequent reasons include: suspected operational problems; significant changes 
planned or underway; expiring or proposed legislation; plans for a major 
initiative; programs or issues the Secretary wants to explore. The SDA teams 
seek especially to identify operational improvements which the Secretary and 
program officials can make without the need for legislative, regulatory, or 
budgetary changes. In this fashion, the SDA findings and recommendations can 
be implemented immediately. 

In the Department of Education, SDA’s will be conducted under the overall 
supervision of OEPM with regional staff providing onsite support necessary to 
conduct the studies. It is hoped that State and local education staff will 
eventually participate with ED regional staff in planning and executing these 
service delivery assessments. 

5, Management Analyses 

Management analyses are studies intended to Improve quality control and 
productivity in Department management and program operations. Improving 
overall productivity is viewed as a crucial responsibility for top-level career 
management in the Department, Recently, management analyses have led to 
recommendations intended to increase efficiency and reduce costs in three 
areas: travel; personnel actions; and student financial aid. 

The emphasis in all three areas is on making accountability and quality control 
integral parts of these processing and delivery systems. Scholarship aid, travel 
documents, or personnel papers do not result from a series of discrete 
managerial tasks but rather are produced by an interdependent system where 
efficiency or inefficiency at any one point affects the entire process. This 
comprehensive perspective fosters accountability by allowing every individual to 
share responsibility for the efficient delivery of services. 

Management analysis also entails the study of the impact and applicability of 
technology, particularly in an office setting. Research and development efforts 
in office automation attempt to avoid the common pitfall of assuming that new 
equipment by itself will increase productivity and efficiency. New equipment 
may merely hide existing administrative problems. Therefore, management 
analysts first examine existing office operations and endeavor to make these as 
productive as possible. The analysts may subsequently recommend that new 
equipment be added, but only based on sound productivity goals and a thorough 
understanding of existing office capabilities. 

6 . Quality Assurance Studies 

Quality assurance studies are intended to assess the accuracy and reliability of 
program data and information used for management decisions. Multidisciplinary 
teams composed of statisticians, economists, systems analysts, management 
analysts, and attorneys use a variety of evaluation and statistical techniques^ to 
assess departmental support systems and programs. Three quality assurance 
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studies currently in progress focus on; grantee monitoring; the process of 
evaluating discretionary grant applications; and developing consistent program 
descriptions. 

Previous studies conducted by the Office of Education revealed the absence of 
an agency policy with respect to grantee monitoring. These studies indicated 
that there were no minimum standards underlying monitoring efforts nor any 
mechanism in place to encourage consistency in monitoring. These studies 
contained a variety of recommendations to resolve these problems. The Office 
of Evaluation and Program Management's Quality Assurance Division is currently 
using these recommendations as the basis for the development of directives, 
manuals, and other policy documents related to the monitoring of State formula 
and discretionary grant programs in ED. 

A quality assurance survey is also underway of all discretionary grant programs 
in the Department (approximately 80) that use nongovernment experts to review 
grant applications. The survey and subsequent data analyses are intended to 
gather baseline information on existing procedures and then suggest ways for 
improving the system's shortcomings. The survey explores topics such as; 1) the 
review process stage at which outside/inside reviewers are used; 2) the use and 
consistency of evaluation standards that reviewers follow; 3) the legislative 
requirements for using grant application reviewers; and 4) the alternatives to 
using nongovernment reviewers. 

The Office of Education did not have an automated system for providing 
consistent and reliable responses to frequently asked questions about education 
programs. To remedy this situation, its quality assurance staff is developing a 
Program Profile System (PPS), a computerized information retrieval system 
containing common key descriptors on each Department program. PPS is being 
designed to generate indices of programs by descriptors as well as to generate a 
listing or profile of all descriptors for each program. PPS will be developed in 
phases; Phase I will profile programs by the Catalogue of Federal Domestic 
Assistance number on 18 descriptors; Phase II will profile subgrant-making 
activities and contract programs; and Phase III will add new descriptors for each 
program. 

7. Organizational Development Activities 

Organizational development activities include the creation of an administrative 
communications system, and work measurement and organizational effectiveness 
studies. The administrative communications system is departmentwide. It is 
used for coordinating the development and distribution of internal management 
directives. Secretary-level circulars, administrative announcements, and the ED 
organization manual. The purposes of the system are to; 1) assure that affected 
parties have ample opportunity to participate in formulating permanent 
administrative policies and procedures; 2) provide complete, concise, current, 
and authoritative information to agency personnel; 3) produce easy-to-read and 
understandable documents; 4) group in a single source for each specialized 
audience information on specific subjects; and 5) make information easy to 
locate. 

Work measurement studies collect current data for a departmental workload 
reporting system that statistically compares personnel resource needs on the 
basis of major output indicators. Within each general category of programs 


13 




{State plan, formula, or discretionary programs), the following major indicators 
are computed for each program: number of applications received; number of 
grants made; number of dollars appropriated; and number of onsite reviews 
conducted each year. The system will assist top management in assessing 
workload trends and identifying potential areas for redistribution of positions. 
Other work measurement studies can be integrated with management 
improvement efforts in such areas as objective-setting, quality assurance, and 
system design. 

Organizational effectiveness studies assist individuals and ED units in working 
more effectively and productively. Projects are designed to examine such issues 
as diagnosing organizational probiems, clarifying communications, establishing 
goals, working with conflict, and improving meetings, decisionmaking, and 
interpersonal relations. 

The services of the organizational development staff include: 1) leadership 
transition assistance to aid new leaders and key management staff to take 
command quickly and work effectively with the bureaucracy; 2) organizational 
diagnosis studies and feedback sessions; 3) consulting on agency trouble spots; 
and 4) designing and conducting tearn-building activities for leaders, managers, 
and their staffs. 

Developing a Systematic Approach to Evaluation 

Having developed an evaluation planning model, continued the conduct of 
traditional process and impact evaluations, and instituted new, shorter term 
evaluation techniques, the Office of Evaluation and Program Management has 
reshaped the evaluation function and operation in the Department. The 
evaluation planning model helps predict information needs at given points during 
a program's life cycle. A wider range of evaluation techniques can now be 
systematically selected with specific program data and information requirements 
in mind. Finaliy, this approach can lend itself to better planning for information 
collection activities other than evaluation studies; e.g., research, statistical 
surveys, and planning studies. 

As an example of how these evaluation techniques and the planning model can 
assist ED officials in obtaining timely and useful evaluation findings, take a 
hypothetical, newly authorized postsecondary student assistance program which 
might require the following types of information during its life cycle. 

0 clarification of program objectives and performance measures 
through either the program objective-setting process or an 
evaluability assessment; timing : planned shortly before final 
authorization and executed immediately after the law is enacted. 

0 design of a delivery system for student assistance using manage- 
ment analysis, organizational development, and quality control 
expertise; original inclusion of a quality control 
system giving feedback information to managers on how well the 
program is working is especially critical at this point; timing: 
planned before final authorization and executed shortly after 
enactment of the law; the delivery system is dependent 
upon perceived program objectives and should incorporate perfor- 
mance measures as part of its management information system. 
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o special information will be needed for annual budget reviews or 
upon request of Department officials; to meet those demands, 
shorter term evaluation tools such as service delivery assess- 
ments, program management reviews, organizational studies, work 
measurement reviews, and management analyses can be planned 
during the program's life cycle; timing ; usually after the program has 
been in operation for 1 to 2 years. 

0 major issues surrounding a program may require larger evaluation 
projects, such as those dealing with effects of Federal funds on 
State and local education agencies, assessment of the national 
strategy used for implementating a specific program, or 
the national impact of a program on the educational achievement of 
children; timing ; depends upon v/hen the information is needed — 
for example, a national impact study whose results are needed for the 
next reauthorization cycle should report its findings approximately 
2 years before reauthorization so that Department officials will have 
the findings as they draft proposed legislation. This, in turn, means 
that the study should begin 3 to 5 years before the reporting 
date so as to have enough time to collect and analyze the data. 

An illustration of a typical evaluation planning model that might be developed 
for such a newly authorized program is shown on the following page. 

Dissemination of Evaluation Findings 

A diverse audience awaits the results of ED evaluation activities. The most 
directly concerned are policymakers and program managers within the 
Department and the congressional committees which have jurisdiction over 
education programs (principally, the House and Senate Appropriations 
Committees, the House Education and Labor Committee, and the Senate Labor 
and Human Resources Committee). Local and State education officials, 
education-related public interest groups, parents, and a sizable community of 
evaluators show varying degrees of interest depending upon the evaluation in 
question. Communicating evaluation findings in a timely and useful fashion to 
responsible officials and an interested education community is critical to the 
success of the overall evaluation effort. 

ED has adopted several strategies to ensure that its evaluation results are 
disseminated rapidly to influence subsequent program decisions or inform the 
education community. One new approach for reaching the broader education 
community with useful evaluation reports is this general circulation volume 
which, used in conjunction with volume 11, provides a comphrehensive overview 
of ED programs, the evaluation process, important new findings, and evaluation 
uses. 

Several alternatives have been developed to assist the dissemination process 
within the Department. A new internal review procedure guarantees all key 
Department officials an opportunity to review and comment on the results of an 
evaluation study within strict time limits. The entire review is completed within 
a short period of time; comments or objections from reviewing officials may be 
incorporated into the final executive summary of the study. 
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EVALUATION PLANNING MODEL 



REQUIRES EVALUATION TECHNIQUES SUCH AS QUALITY CONTROL, MANAGEMENT ANALYSIS, AND ORGANIATIONAL DEVELOPMENT 



Regular staff briefings on evaluation findings can also be an effective 
mechanism for communicating evaluation information within the Department. 
For example, OEPM offers to brief each program office on the budget 
implications of evaluation findings. The budget briefing is keyed to the 
beginning of the annual budget-preparation cycle. The evaluation staff also 
meet with program officials to discuss the implications of evaluation findings on 
appropriations and legislative reauthorization hearings before congressional 
committees. 

A newly established Program Resource Center within OEPM represents another 
way in which information on programs administered by ED will be made available 
to members of the Department. The center will collect and index all 
management and program evaluations conducted by ED or other Federal agencies 
(e.g., the General Accounting Office, the Congressional Budget Office) that 
pertain to Department programs or operations. Other program documents will 
be added as staff resources and time permit. At a minimum, the resource center 
will maintain copies of all final evaluation reports and executive summaries and 
other key evaluation documents referenced in the current edition of the Annual 
Evaluation Report . Eventually program files could include legislation, planning 
studies, policy analyses, program regulations, program guidelines, and annual 
program reports to the President, the Congress, or the Secretary of ED. Plans 
are underway to have as many documents as possible recorded on microfiche or 
magnetic tape for convenient access and long-term storage. 

Efficient use of evaluation findings has become a top management priority in 
ED. Communicating findings quickly is an effective, indirect way of achieving 
this objective. A more direct approach for evaluation followup has also been 
developed on an experimental basis. When an evaluation report is likely to have 
a bearing on program legislation, budget decisions, major program operations, or 
management practices, the Office of Evaluation and Program Management, in 
cooperation with the Assistant Secretary for Planning and Budget and the 
appropriate program's Assistant Secretary, will establish an Evaluation 
Implementation Board. The Board, consisting of senior evaluation, planning and 
budget, legislation, legal, and program staff, v/ill review the evaluation report 
and synthesize its findings into a decision memo for the Secretary, Under 
Secretary, or appropriate Assistant Secretary. The memo will contain policy- 
relevant and management improvement options and recommendations. Once 
decisions have been made, the Board will also develop an implementation time 
schedule. 


CONCLUSION 

Since the Department's creation, the evaluation function has changed direction. 
The scope of evaluation interest has been broadened to include all aspects of 
program operations. Program impacts are assessed as part of a complex process 
that includes ongoing program operations, budget, and management activities. 
Innovative management and evaluation techniques are now in place beside 
traditional evaluative methods. Greater attention is paid to systematic 
evaluation planning and comprehensive evaluation followup. The overall result is 
a flexible, responsive, and useful evaluation capability within ED. 
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NOTES 


^These activities are discussed later in this chapter, 

^Two evaiuability assessments that were begun in FY 1979 under the 
Department of Health, Education, and Welfare have been terminated. Managers 
of the Vocational Education Program and the National Center for Education 
Statistics asked that the assessments be stopped because the heavy workloads 
imposed by the transition into the Department of Education prevented their 
participation. Because of organizational and staffing changes which have since 
taken place, it would be necessary to start new assessments rather than attempt 
to finish either of these studies. 

^They include; validation and measurement techniques; statistical quality 
control charts; regression analysis; and sampling techniques. 
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Chapter II 


Education Program 
Overviews 



This chapter is an overview of ED program activities and evaluation findings that 
highlight program operations. The underlying theme of the following materials is 
diversity — diversity in the types of programs ED funds, intended recipients, and 
the mechanics of program operations. Not only do programs vary widely, but 
occasionally so do implementation procedures within a single program. Nearly 
all variations result from the flexibility in program operations that the Congress 
mandates; States and localities are expressly given latitude in program 
implementation. In addition, the operations of State and local education 
agencies and universities differ enough that even a common legislative goal may 
translate into substantially different procedures and, perhaps, even results. 

Understanding the diversity between ED programs, and occasionally within them, 
is one of the principal objectives of the Department's evaluation activities. It is 
also one of the greatest challenges because few definitive answers or 
generalizations result from this work. As will become evident in subsequent 
sections of this chapter, conclusions from even the most thoroughgoing and 
comprehensive evaluation studies are often reported with caveats. Caution is 
usually in order because of the changes expected to result from the influence of 
ED-funded programs. Overall educational achievement, for example, is 
influenced by a multitude of factors including unique individual, family, and 
community characteristics. Many of these factors cannot be measured; others 
cannot be controlled statistically. 

While caution is clearly called for in Interpreting broad educational change as a 
result of program activity, fairly reliable assessments of immediate program 
impact are possible. Subsequent sections describe such assessments. The 
elementary and secondary school overview discusses programs for low-achieving 
students, handicapped by poverty or a limited ability to speak English. Recent 
evaluation findings indicate that while Federal education funds have reached 
these intended recipients more effectively in recent years, there is still room for 
improvement. Progress has also been made in program implementation. 

Program impact on improving the academic skills of elementary and secondary 
school students is seen to vary by subject matter and grade level. 

The overview of postsecondary education programs focuses on enhancing 
educational opportunity by providing Federal financial support to students and 
institutions. Over time, significant changes and differences are found in 
attendance at institutions of higher education by students' sex, family income, 
and race. The goals of Federal grant, loan, and work programs are to improve 
student access to and persistence in higher education and provide a range of 
institutional choices. This section assesses the degree to which Federal funding 
has succeeded in achieving these goals by utilizing the concept of student 
burden—the difference between the cost of attending a particular college and 
Federal aid received. Federal aid to institutions aiso affects students' 
postsecondary school opportunities. These influences along with student transfer 
patterns among higher education Institutions are briefly reviewed in this section. 

The third and final section of chapter 2 is an overview of the large and diverse 
number of education programs directed toward special recipient groups or 
particular educational needs. These include programs to improve the quality of 
educational practices; expand learning resources; foster occupational, career, 
and adult education; and assist the handicapped and Native Americans In 
receiving quality education. The results of evaluation studies in this section are 
as wide-ranging as the programs themselves. One general statement is 
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applicable however: funds for evaluating these special category programs have 
been extremely limited. A few have special evaluation monies (set-asides) 
authorized and appropriated by the Congress . Most do not. Given the relatively 
modest appropriation for many of these programs and the Department's 
extremely limited discretionary funds, many programs have not been formally 
evaluated. For these reasons, special category programs will benefit 
significantly from the new management improvements initiated in the 
Department and discussed in chapter 1. 


ELEMENTARY AND SECONDARY EDUCATION PROGRAMS 

Approximately 48 million students are currently enrolled in elementary and 
secondary schools. About 90 percent attend the 87,000 schools operated by 
nearly 16,000 public school systems; the other 10 percent attend non-public 
schools. 

Many educators and the general public have expressed concerns about the 
conditions in our public school systems. According to an annual Gallup Poll, the 
public's rating of schools declined through the latter half of the seventies, but 
this decline may have come to a halt in 1980. Although only 28 percent of the 
public has a "great deal" of confidence in the public schools according to that 
poll, the standing of public schools exceeds that of all other important American 
institutions except churches. 

The five major problems confronting public schools, as perceived by the poll’s 
respondents, are; (1) lack of discipline; (2) use of illegal drugs; (3) poor curricula 
and standards; (4) lack of proper financial support; and (5) integration or busing. 
Lack of discipline was rated as a problem almost twice as often as any other 
topic. 

Other public opinion polls register widespread concerns about education. But for 
many of the problem areas cited, it is difficult to describe the exact nature of 
the problem, let alone express the degree to which the problem is real. Measure- 
ment is relatively good, however, for educational achievement as reported by the 
U.S. Department of Education's National Assessment of Educational Progress 
(NAEP). Every few years NAEP assesses the educational achievement of 9-year 
olds, i3-year olds, and l7-year-olds in major academic subjects. 

Most subjects were measured twice during the 1970*s and, at the risk of some 
oversimplification, can be summarized for reading, math, writing mechanics, 
writing quality, science, social studies (9-year olds) and politics (teenagers) as 
follows: 

o 9-year olds (mostly fourth graders) showed slight improvements in 
reading, slight declines in math and science and not much change 
elsewhere. 

o i 3-year olds (8th graders) and 17-year olds (12th graders) showed 
varying degrees of declines in knowledge of math, science, and 
politics, and showed essentially no change in reading and writing 
mechanics. 

Part of the public's concern about education may reflect the measured declines 
in performance by teenagers. 
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Major Federal education programs are aimed at just a few of the national 
concerns about elennentary and secondary education. The issues to be discussed 
in this section are education for low achievers (especially when achievement is 
related to low family income or limited English proficiency), school 
desegregation, and general improvement in educational practices. The primary 
questions are: (1) how well are Federal education funds getting to the intended 
beneficiaries; (2) how well have the programs been implemented; and (3) how 
effectively have the programs accomplished their objectives? 

How Well Are Federal Education Funds Getting to the Intended Beneficiaries? 

The efficiency with which Federal funds are targeted is indicated by two 
measures: the proportion of ineligible program participants and the proportion 
of eligibles not served. The latter measure is complicated by the fact that the 
percent of ellgibies served is determined not just by the efficiency of the 
targeting process but also by the total funds available. If funds are insufficient, 
it is impossible to serve all the eligible population. 

For Title I of the Elementary and Secondary Education Act (ESEA), extensive 
data pertaining to targeting are available for children in grades 1-6, grades where 
local school districts spend a high proportion of Title I funds. Results from the 
1976-77 school year show that among studei^t's scoring below the median on 
achievement tests, 25 percent (approximately 2.5 million students) participated 
in Title I. Of children scoring above the median, only percent participated in 
Title I. These results indicate that Title I is fairly well targeted on the low- 
scoring students but that many low achievers (7.3 million students) below the 
median do not participate.! 

Although the language of Title I seems to stress meeting the needs of low 
achievers, some hold that the funds should also be focused on children from low- 
income families. Moreover, a high correlation is often assumed between low 
achievement and poverty. The data indicate, hov/ever, that the correlation (r) is 
modest, ranging from .2 in the first grade to .3 in grades three to four. As a 
consequence, even though Title I funds are fairly well-targeted on low achievers, 
about 58 percent of the participants are not from poor families (based upon the 
Federal Government's poverty indicator). ~ 

The Title 1 migrant subprogram offers special services for the children of 
migrant workers. A Migrant Student Record Transfer System (MSRTS), using 
data supplied by school systems, estimates the number of program participants in 
1979 at 522,000. The size of the eligible population is unknown, but it is believed 
that not all are served. There are probably two categories of unserved children: 
first, some unknown number of migrant children are seldom, if ever, enrolled in 
school; second, from 15 to 20 percent of eligible children enrolled in school are 
not registered on the MSRTS. ^ (An unknown percent of such children are 
probably not served.) On the other side of the targeting question, probably fewer 
than 2 percent of participating children are ineligible. 

Some 51,000 Title I-eltgible children are in State institutions for neglected or 
delinquent children, but 1976 data indicated that only 27,000 (53 percent) were 
receiving services. No estimate is available for the number of participating 
ineligibles. In both the migrant and neglected/delinquent programs, the 
differences between the numbers of children served and eligiWe can largely be 
attributed to targeting inefficiencies (not lack of funds) since programs are fully 
funded. ^ 
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In contrast to Title I, the ESEA Title IV-C Program (Support and Innovation 
Grants for Improving Local Educational Practice) is not aimed at a particular 
target group. However, the legislation requires that children attending nonpublic 
schools must receive program benefits on an equitable basis with public school 
students. Recent evidence indicates that nonpublic students are participating in 
programs funded by Title IV-C grants on a less than equitable basis. On the 
other hand, the majority of eligible nonpublic schools, and presumably their 
students, receive Title IV-B (Instructional Materials and School Library 
Resources) benefits for learning materials and library resources. 

ESEA Title VII (Bilingual Education) is primarily for children of limited English 
proficiency. Though the program is funded at an amount greater than a 
demonstration program, it is substantially below the level required to serve all 
eligible children. Therefore, the percent of eligibles served is not a useful 
indicator of targeting efficiency. However, the percent of English-proficient 
childreii participating in the program is an appropriate indicator. 


A 1978 study of Title VII found that less than one-third of participating students 
were of limited English proficiency. Although there was no legal restriction 
against participation by English-proficient children, a proportion of more than 
two-thirds was widely felt to be too high, and the Congress amended the law in 
1978 to limit participation of English-proficient children to ifO percent. There 
has been no systematic check of the language proficiency of participating 
children since that change in the law. 

In the Emergency School Aid Act (ESAA), the targeting issue pertains more to 
school districts than to schools or children. The purpose of the program is to 
encourage the voluntary elimination, reduction, or prevention of minority group 
isolation and to meet special needs incident to the elimination of segregation. 
Several studies have found that most funds for the basic grants subprogram went 
to districts that desegregated long ago. In response to this situation, the 
Congress recently amended the law to add recency of desegregation to the 
criteria for judging grant applications. 


How Well Have the Programs Been Implemented? 

It is by now widely acknowledged that social intervention programs are often 
poorly implemented. Federal programs in elementary and secondary education 
have not been exempt from this problem. It is also probably true that many 
evaluation studies of educatioh programs have not given sufficient attention to 
assessing implementation.^ Inadequate technology (e.g., lack of instrumentation, 
data collection problems) and high costs are the main reasons for limited 
measurements of the degree of implementation* 

Nevertheless, most evaluation studies do provide some information about 

® example. The 1965 version of the law 
Congressional intent. Initial direction from the 
thaTmiSh^iariv^ Education was similarly loose. It is not surprising, therefore, 
and actions fnrLrtfr fo^^^sed on possible miluse of funds 

howevirbo h th. . Congress. Over the years, 

nowever, both the law and program administration have become more rigorous. 

ShooMittHr?''® that, on the whole, the Title I grant program to 

school districts is reasonably well implemented. Following the apparent intent 
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of the Congress, local projects tend to emphasize instruction in basic cognitive 
skills. Generally, Title I services supplement the usual services provided to 
disadvantaged children, and in amounts large enough to be clearly noticeable as 
an educational program distinct from that offered regular students. Much less is 
known about the proficiency of project management and the quality of 
instruction. 

The Congress seems to have intended that the Title 1 subprogram for 
institutionalized neglected and delinquent children be similar in purpose and 
operation to the district grant subprogram. A recent study indicates that the 
projects do emphasize instructional services and that the Title 1 funds are indeed 
supplemental to State funds. Otherwise, there seems to be wide variation in how 
institutions implement the program. Actual receipt of services (as measured, for 
example, by hours of reading and math instruction) is less, on the average, for 
Title I students than for non-Title 1 students. The difference seems to exist 
because some institutions do not attach much importance to student attendance 
in the special Title 1 activities. 

The implementation question is far different for a program such as ESEA's Title 
IV-C. Here the keynote is State and local flexibility, and the Congress has 
imposed very few constraints. A recent evaluation shows that management of 
the program has fostered flexibility. As a result, there is great variation among 
the States with respect to program administration and types of projects. 

However, the program did not, in general, lead to consolidated management of 
formerly categorical programs as may have been hoped. 

The National Diffusion Program (NDP) takes a different approach toward 
improving school practice. With this relatively small program, a number of 
educational practices have been identified for which there is evidence of 
effectiveness. Evaluative evidence indicates that the National Diffusion 
Program has been successful in causing many school districts to adopt exemplary 
educational practices. Further, there is evidence that districts frequently adapt 
the practices to local circumstances. At this time, it is not possible to say with 
certainty whether good NDP implementation should be defined as high-fidelity 
adoptions with few local modifications, or vice-versa, or something in-between. 
Indeed, the answer may depend upon the kind of educational practice in question. 
Until there is further evidence on the effectiveness of adoptions, the question 
will be not answered. 

The Emergency School Aid Act. (ESAA), and especially the basic grants 
subprogram, exemplifies that program implementation depends upon the clarity 
of congressional intent as expressed in legislation. Results of several studies 
show that a high proportion of the basic grant funds are used for general 
education assistance or compensatory education rather than for desegregation 
assistance, which is the program intent. Although not an indication of illegal 
operations, these findings highlight the loose link between the program purpose 
and the actual use of funds. In 1978, the Congress amended the ESAA legislation 
to encourage a closer relationship between the program purpose and the 
activities funded. 

How Effectively Have Programs Accomplished Their Objectives? 

There is rarely, if ever, complete agreement among all members of the Congress 
or all administration officials about the objectives of Federal education 


25 



programs. The following results pertain to the most frequently stated program 
objectives. 

Schools tend to focus ESEA Title I funds on reading and math instruction, 
especially in the elementary grades. Recent evidence (students tested in 1976- 
77) shows that for student reading achievement in grades 1, 2, and 3, Title I is 
effective in that Title I students learn to read at a rate faster than would be 
expected without Title I but not fast enough to catch up with regular students. 

In grades 5, and 6, Title I does not impart benefits in reading. Math results 
show positive Title I effects in grades 1 through 6. The details of these results 
are rather complicated and the reader is advised to consult specific reports cited 
in the Title I section of volume 11. 

Among students in Title I projects for neglected or delinquent children (students 
tested in 1977-78), the reading and math scores were about what would be 
expected in the absence of Title I; that is. Title I is not having an impact on 
these students in those skill areas. 

Information about the ESEA Title VII program comes from a study (students 
tested in 1975-76-77) of Spanish-English projects in their fourth or fifth year of 
operation. In English Language Arts, achievement gains of Title VII Hispanic 
students were less than those of non-Tltle VII Hispanic students. In math, the 
gains of the two groups were comparable. 

An early (1973-76) longitudinal evaluation of the ESA A basic grant program found 
evidence of improved academic achievement in elementary sctiools but no 
improvement at the high school level. Emphasis subsequently shifted away from 
improved achievement as a program objective and toward improving relations 
among students. A recent study (data collected in 1978-79) found that students 
who received ESAA-funded human relations services showed greater 
improvement in measures of intergroup relations (e.g., attitudes, behavior, and 
self-concept) than students without such services. Evaluations of two other 
ESAA subprograms showed mixed results. One study found that ESAA Magnet 
Schools can be effective when used as a component of a comprehensive, district 
wide desegregation effort and when used in districts with a relatively easy 
desegregation task. Another study showed that organizations receiving grants 
from the ESAA Nonprofit Organization's subprogram were less effective in 
developing community support for desegregation than community groups not 
funded by ESAA. 

Federal programs aimed at general improvement in educational practices are 
difficult to evaluate because of the diversity of practices and the sometimes 
intangible changes being sought. Thus, while it is clear that ESEA Title IV-C has 
encouraged State and local education agencies to try alternative educational 
practices, no overall statements can be made about the impact of those changes. 
Likewise, for the National Diffusion Program which seeks change in a more 
direct and specific way than Title IV-C, broad judgments can not be made 
regarding program impact on students. 

Summary 

An important role of program evaluation is to inform policymakers about 
program performance in targeting, implementation, and effectiveness. As 
studies are completed, evidence about each aspect of program performance 
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increases and so does knowledge about how to implement necessary program 
change. It is clear that conclusions about Federal education programs in 
elementary and secondary schools are mixed; in some cases (such as district 
grants of ESEA Title I) funds appear to be reaching the intended beneficiaries, 
providing services as outlined in law and regulations, and leading to modest 
accomplishment of overall goals. In others, the funds may be targeted 
appropriately but do not yet lead to services well focused on progam 
participants. For these programs the effectiveness of program services is still 
largely untested. 

Finally, an occasional evaluation in elementary and secondary education has 
found that participants in the program are not those identified as the needy 
group, with the result that policymakers in the Congress and the administration 
have had to clarify eligibility criteria (e.g., ESEA Title Vll). Of course, this 
means that both implementation and effectiveness in Title VII need to be 
examined (or re-examined) since the characteristics of participants have 
presumably changed. 

Individual elementary and secondary programs covered in volume II of this report 
offer examples of each type; programs which appear to be targeted 
appropriately, implemented well, and are effective; those which are well- 
targeted but need more attention in the area of implementation; and those whose 
target group has been modified based on findings of earlier evaluation studies. 


POSTSECONDARY EDUCATION PROGRAMS 

The primary goal of the Department of Education's postsecondary education 
programs is to increase educational opportunity. The strategy for achieving this 
goal rests 

principally on providing financial assistance to both students and institutions. 
Federal student assistance is awarded both directly and indirectly through State 
agencies and education institutions. In fiscal year 1980 the obligations for ED's 
student financial assistance programs totaled almost $5,2 billion. Institutional 
assistance is granted primarily through the Developing Institutions Program and 
special programs for disadvantaged students. In fiscal year 1979, institutional 
assistance amounted to over $400 million. 

Participation Rates 

Figures on college enrollment for 18- to 24-year olds during the 1970-78 period 
are presented in table 1. Participation rates for various subgroups tend to be 
more volatile than for the age group as a whole.^ Over this period, participation 
rates for males have declined while those for females have Increased. In 
contrast to more or less constant participation rates for whites, participation 
rates for blacks and Hispanics have increased markedly. 

Table 2 presents the annual enrollment rates of dependent students since 1974 
categorized by family income (measured in current dollars). Over this 5-year 
period, a clear trend of declining enrollment rates in the over $15,000 family 
income categories emerges for both males and females. This pattern contrasts 
with the enrollment rates for dependents from lower income families, which 
have remained relatively stable during this period. 
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Table 2 

PFRPFNTAGE OF 18-TO 24 -YEAR-OLD DEPENDENT FAMILY MEMBERS ENROLLED IN COLLEGE, 
^ BY SEX AND FAMILY INCOME 


00 ^ s:}- C\J LO ^ 

cTi ro LD 00 ro cvj 00 

r- f— I — f— OJ CO 


VO CM VO * — cr> o 

VO LDCMr-- 

r- I— r— CM CO ^ 


oo CO CM VO 

« 4 • t • • 


o c^ CM cn 

r— I — r-- CM CO 


e'- 

en 


r- m CO *51- CM VO 

CO lO VO CO ^ 
I— f— r— CM CO Lf) 


<h " 5 :^* CM r— O 

^ r-- o r-- o 

r— CM CM ^ VT) 


O 

UJ 


O 

O' 


VD 

CO J — 

^ CO 

«— CM 


a a 

« • 

1 • 

Qi 

^ LO 

CJ 

^ lO 


r— 

CM CO 

^ lO 


CM <— VO r— 00 cr> 

r- CM CO •— CM 
I— I — CM CO ’vf' VO> 


LiJ 


CM 

1 

oa 


■p 

c 

<D 

o 

t- 

a> 

O- 


Lf) 

c- 

cn 


CO VO CM r— 

# • ♦ • • • 


4 ;^* LO CM cn CO 

»— r- CM* CM «:t ur> 


o oO CM 

• • • • • * 

VO c-- ^ CO VO 

I— r- CM CO LT) 


<=t- 

r-- 

cn 


o cn uo CO 00 

• • « « « • 

CO ^ O O CO CO 
t— I— CM CO «Cf VO 


CM O C-- CM h-- • — 

• t • • * « 

VO 00 CM CM ^ CXD 
r— 1 — CM CO ^ vr> 


o 

B 


o 

o 

c 




cn CJT cn cn cn 

C7i CT^ CT^ cr> 
CTi OY CT» 

fS «v 


cn ^ 


I 

o 


cn ^ 

r- CM + 

till 

o o o o o 
o o o o o 
o o o o o 


VO o Lo o vn 

«— * r— CM CM 

■to 


cr> OY (Ti CTy 

CTi OY CT* CTy CJY 
on OY CTi CTk a> 


1 

o 


CT> «c:}- cn *:d- 
I — I— CM “h 
1 1 t I 

o o o o o 
o CO o o o 
o o o o o 

^ r» rv ri «■ 

ID o un o vn 

r— r- CM CM 


-bO- 


I— o "sf C'' ro CO 

«. • • •• • • 

CM ^ 00 VO VD LO 
r— t— CM CO tO 


VO lo «d” lo ^ cn- 

CM CO O CM VO 
I — r- CM CO *==*' LO 


rOVO^-CM<s^'*— 

« • a • • 4 

aef to CM <Ti r— CO 
I— CM CM ^ lO 


VO •sd" 00 VO 
o CM 00 00 I — CT» 

t— c— I— CM LO 


CJY CTs QY C3% 

CTi cty OY cty cn 

cr» CTY on cTk 

n at rk a> 


CT> ^ CJY 

r-r*-CM + 
l 1 I I t 

o o o o o o 

o o o o o 

o o o o o 


vn O LD o VO 
J— ‘ r— CM CM 

-bO 


o 

CM 

I 

O- 

c/1 

CD 

»r~ 

u 

CD 

LTi 


in 

4-> 

u 

o 

CM 

<D 

a: 

c. 

o 

*r- 

4-> 

ftJ 

04 

o 

Cl- 


c 

CD 

su 

13 

o 


1/1 

r3 

V/1 

c 

QJ 

o 


o • 

VO 

« CO 

d) 

V- 

=3 CO 
Cd CO 
CO 


CO 

• cn 

*— 

E ^ 

O 

S- CO 
Vfr- O 
CO 

-o 

<u ^ 

4-> VO 
«3 00 
I— CM 
3 

L> * 

r— t/1 

CO o 
o s: 


o 

ct: 

::d 

o 

LO 


29 



The patterns do, however, appear to be consistent with a general decline in the 
long-run financial benefits accruing to a postsecondary education — a decline 
that is mitigated for certain population subgroups (minorities and the poor) by 
transfer payments in the form of student financial aid that is conditional upon 
attendance. 

The preceding discussion has focused exclusively on college enrollment because 
no comparable time-series data exist for the noncollegiate sector of 
postsecondary education — proprietary and public technical/vocational schools. 

Improving Access and Choice Through Student Support 

Enhancing educational opportunity through student financial aid programs 
removes financial barriers to a postsecondary education for qualified applicants 
(access) and provides each potential student with as wide a variety of 
postsecondary education options as possible (choice). A number of assumptions 
underlie this general position. They are: (1) that parents bear the primary 
responsibility for financing their children's education; (2) that students 
themselves bear some of the burden of financing their education; (3) that the 
portion of the financial burden borne by the students themselves be distributed 
as equitably as possible; and (^f) that the most needy students be aided first. 

ED-administered programs provide three types of student assistance. Grant or 
non-returnable aid is provided by the Basic Educational Opportunity Grant 
(BEOG), Supplemental Educational Opportunity Grant (SEOG), and State Student 
Incentive Grant (SSIG) programs. Obligations for these programs totaled $2.8 
billion in fiscal year 1980. Loans are provided by the Guaranteed Student Loan 
(GSL), National Direct Student Loans (NDSL), and Health Education Assistance 
Loan (HEAL) programs. The 1980 obligations for these programs amounted to 
$1.9 billion.^ Earnings are provided by the College Work-Study (CWS) Program. 
Obligations for CWS were $550 million in fiscal year 1980. 

The grant programs (as opposed to the loan and work programs) have come to 
serve distinct roles in removing financial barriers to access and increasing 
choice. Grants are seen as providing the financial support that less well-off 
parents are unable to provide. Loans and work-study jobs, on the other hand, 
offer students the means to share the financial burden of attending an 
educational institution. Offering the students both loan and work opportunities 
allows them a choice in the timing of when they ultimately bear their share of 
the financial burden. Work allows them to bear it concurrently with their 
schooling; loans allow them to put off the burden until after they finish their 
education. 

Table 3 displays the proportion of undergraduate students enrolled on at least a 
half-time basis who were aided by one or more of ED's student assistance 
programs (BEOG, SEOG, NDSL, GSL, and CWS^) categorized by dependency 
status, family income, and category of institution attended. 

In academic year 1978-79, 36 percent of all undergraduates who were enrolled on 
at least a half-time basis were receiving some student assistance from ED. Of 
those attending il-year public Institutions, 36 percent were receiving aid; of 
those attending ^f-year private institutions, 45 percent were receiving aid; of 
those attending 2-year public institutions, 26 percent were receiving aid; of 
those attending 2-year private institutions, 37 percent were receiving aid; and of 
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Table 3 


Percentage of Undergraduates Enrolled at Least Half-time Receiving 
Aid from at Least One of ED Student Assistance Programs 
{Unduplicated Count) 1978-79 


By Family Income Group Percent 


Dependent 

$ 0- 5.999 

76 


6,000-11,999 

67 


12.000-17,999 

47 


18,000-24,999 

27 


25,000-29,999 

21 


30,000 or more 

8 

Independent* 


37 

All Students 


36 


By Institutional Category 


4 - Year Public 36 
4 - Year Private ' 45 
2 - Year Pub! ic 26 
2 - Year Private 37 
Proprietary 63 


Note: These include only the Basic Educational Opportunity Grants, 

Supplemental Educational Opportunity Grants, National Direct 
Student Loan, Guaranteed Student Loan and the College Work- 
Study Programs; the State Student Incentive Grants Program is 
not included. 

Source: "Study of Program Management Procedures in the Campus-Based 
and Basic Grant Programs - Stage 2", Applied Management 
Sciences, Silver Spring, MD; 1980. Contract funded by 
Office of Evaluation and Program Management/EO 


* Considered independent of parental support for student and distribution 
purposes. 
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those attending proprietary institutions, 63 percent were receiving aid. Overall, 
a greater proportion of students appear to be aided in the more expensive 
categories of postsecondary institutions. 

As might be expected, among dependent students the proportion receiving 
student aid declined as famiiy income increased. In the lowest income category 
($0 - $5,999), 76 percent of the students received some form of assistance, 
whereas in the highest income category ($30,000 or more) only 8 percent of the 
students received some assistance. Among independent students, 37 percent 
received aid from ED student assistance programs. These percentages are based 
on unduplicated counts of students receiving aid from any of the five ED 
programs. 


Table ^ describes the extent to which undergraduates received aid from more 
than one ED program. BEOG recipients were least likely to receive aid from 
other ED programs; if they did receive other aid, it would most likely be a CWS 
job. The vast majority of SEOG, CWS, and NDSL recipients also receive BEOG 
assistance. There appears to be very little overlap between the loan programs. 
Only 10 percent of NDSL borrowers are reported to also have a guaranteed 
student loan, and only 16 percent of those with a guaranteed student loan are 
reported as having an NDSL. 


Impact of Student Financial Aid Programs 

The achievement of the ’’access” objective of the student financial assistance 
programs can be measured by the absolute size of the financial burden a student 
must bear if he or she is to attend a postsecondary institution. This "student 
burden" is the difference between the cost of attending an institution (tuition, 
fees, room, board, etc.) and the nonreturnable aid the student receives from his 
family, the Government, or other sources (i.e., family contribution, grants, and 
scholarships). Presumably the student burden is financed by some combination of 
borrowing and work activity, and thus for students attending postsecondary 
institutions, it can be measured by the sum of their loans, earnings from part- 
time work, and savings from summer work. 


f measuring the impact of 

* access. ' The first is the size of the student burden. If 

c.irr,^ ^ .Size should not exceed reasonable part-time earnings and 

aspect of the student burden as a measure of program impact is the degree to 
which It IS equal or unequal across need categories, as measured by family 
resources, for different types of institutions (selectiveness, public^or private 2 

hnrw h ^ iijancial opportunity, which is derived from the premise that the 
burden borne by students should be as equal as possible. ^ 

"dMktf''obiectlve°c^^ (inancW aid programs with respect to the 

dr 

degree o£ equality relative burdens serverjTrea“rV:£Th?ertet^^^^^^^^^ 
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Table 4 


Percentage Rates of Program Overlap in Student Aid Recipiency, 
For Undergraduates in the Fall of 1979 


0 Of students receiving BEOG's, 

24% received SE0G*s, 

36% received CWS, • 

25% received NDSL's, 

10% received GSL's. 


0 Of students receiving SEOG's, 

87% received BEOG's, 

50% received CWS, 

48% received NDSL's, 

8% received GSL's. 


0 Of students receiving CVIS, 

82% received BEOG's, 

37% received SEOG's, 

35% received NDSL's, 

14% received GSL's. 


0 Of students receiving NDSL's, 

76% received BEOG's, 

40% received SEOG's, 

38% received CWS, 

10% received GSL's. 


0 Of students receiving GSL's, 

48% received BEOG's, 

11% received SEOG's, 

26% received CWS, 

16% received NDSL* s. 


Source: "Study of the Impact of the Middle Income Student Assistance 
Act," Applied Management Sciences, Silver Spring, MD, 1980. 
Contract funded by Office of Evaluation and Program Management/ED 
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grant programs equalize the financial terms upon which prospective students 
must choose between postsecondary options. Clearly, relative student burdens 
will be equal if the student burden associated with each education option is the 
same regardless of the student's family resources. An education option in this 
context can be defined by the total cost of attendance or the student's expense 
budget. 

Evidence Concerning the Impact of ED's Student Financial Assistance Programs 

Tables 5 and 6 show estimates of the mean expected family contribution, total 
grant aid received, and student burden borne by financial aid applicants, 
categorized by dependency status, family income for dependent students, and the 
cost (table 5) or type of the institution attended (table 6). The estimates of 
mean student burden are useful in assessing the impact of ED's financial 
assistance programs. It should, however, be remembered that these estimates 
are derived from data on aid applicants only. 

A comparison of student burdens with what students can be reasonably expected 
to earn or borrow sheds light on the achievement of both the access and choice 
objectives of the student financial aid programs. As a reference, a student 
working at the minimum wage for 15 hours a week during the school year would 
earn approximately $1,350. Summer work could reasonably produce another $500 
in savings. Therefore, from work alone a student should be able to finance a 
student burden of $1,850, provided, of course, that a job exists. Under the 
Guaranteed Student Loan Program, all students are eligible to borrow up to 
$2,500 per year for 3 years? "needy" students can be afforded loans by 
postsecondary institutions of up to $2,500 in their first 2 years of study and 
another $2,500 in their second 2 years under the National Direct Student Loan 
Program. 

If access is interpreted as adequate financial resources to attend a 2-year public 
institution, the mean student burdens presented in table 6 indicate that the 
access objective has been reasonably accomplished. For all dependent students, 
the student burden could be easily financed by part-time work alone. The 
independent students' burdens are significantly higher but seem manageable if 
students commit much more time to work activities or are willing to take out 
sizable loans. Little change is noted if access is defined in terms of 4-year 
public institutions. Student burdens at these schools are higher for all 
categories, except independent students. Where higher, still easily financed by 
work activity. 

What about choice? How feasible are the student burdens at private and 
proprietary schools? Looking at table 6, we see that the largest mean student 
burden is $3,774 (for independent students at private 4-year institutions). A 
burden of this magnitude is approximately $300 less than the sum of reasonable 
expected student earnings ($1,850) and annual student borrowing ($2,500). Where 
postsecondary education options are defined by their total cost in table 5, the 
largest mean student burden is $5,349 (for independent students at institutions 
costing over $6,000). A burden this large is almost $1,000 more than the sum of 
what a student can borrow and reasonably be expected to earn ($4,350). If these 
students choose to attend a high-cost private institution, they can do so only at 
great personal sacrifice. 



Table 5 


Mean Expected Family Contribution^ Grant ftid and student burden for Aid App^l leant s by Dependency 
Status, by Family Income, and by the Student Expense Rudget of the Institution Being Attended: 1979-80 

(in dollars) 


Dependent Students by Family Income Independent Students 


Student Expense 
Budget 


0 

-5.999 

$6 ,000 
-11,999 

$12,000 

-17,999 

$18,000 

-23,999 

$24,000 

-29.999 

$30,000 
and over 


0. 

») 

Expected Family Contribution 

241 

277 

625 

782 

1.293 

b) 

1.033 

185 

$2,500 

All Grants 

1,097 

1,063 

829 

79ft 

680 

765 

849 


Student Burden 

510 

535 

574 

510 

287 

250 

880 

$2,50) - 

a) 

Expected Family Contribution 

439 

629 

661 

1,015 

1,171 

1,447 

293 

3, ODD 

All Grants 

1,230 

1,096 

1,082 

852 

694 

396 

727 


Student Burden 

1,049 

1,021 

1,020 

869 

891 

808 

1,772 

$3*001 - 

Expected Family Contribution 

302 

370 

729 

1,151 

1,475 

1,645 

225 

4,000 

All Grants 

1,786 

1,670 

1,473 

1,201 

823 

1,018 

948 


Student Burden 

1,058 

1,573 

1,414 

1,229 

1,314 

959 

2,388 

$4,001 - 

Expected Family Contribution 

257 

288 

767 

1,779 

2,665 

2,225 

382 

5.000 

A1 1 Grants 

2,260 

2,271 

1,880 

1.238 

752 

1,002 

1,288 


Student Burden 

2,037 

1,977 

1,902 

1,565 

1.125 

1,348 

2,B41 

$5,001 - 

a) 

Expected Family Contribution 

369 

545 

884 

1,405 

2,026 

3,146 

361 

6,000 

All Grants 

2 612 

2,405 

2,002 

1 904 

1.634 

979 

1 424 


Student Burden 

2,437 

2,593 

2,548 

2,251 

1.960 

1,396 

3,642 

Over - 

a) 

Expected Family Contribution 

860 

823 

1.379 

2,450 

3,012 

4.730 

893 

6,000 

All Grants 

3 293 

3,366 

2,844 

2 611 

2v256 

1 .421 

1.515 


Student Burden 

3,222 

3,186 

3,413 

2,2fl5 

2,202 

1,626 

5,349 


Mote: aT The Expected Famfly ContrlbuTion is computed according to the UnfFonTi Hetho<rdrogy, adju s^teTwhe re 'appro pr fate so 
as not to be less than $1J0D for a stvjdent living with parents. 

b) Cell frequency fewer than 30, 

Source: "Study of the Impact of the Middle- Income Student Assistance Act," Applied Management Sciences, Silver Spring, t-ID 

1980, Contract funded by Office of Evaluation and Program Management /ED. 
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TMe 6 


Mean Fxppf ted Famity Contrfbwtion. Grant ^^<\ and Student Burden for Aid Applicants by Dependency 
Status t by Family Ircornet and by Type of Institution Beiny Attended* 1979-80 

<in dollars) 


Qepende nt Students by Fam ily I ncome Independ ent Stiidrntj, 


Type of 
Institution 

al 

Expected Family Contribution 

0 

-5 ,999 

$6,000 

■11,999 

$12,000 

-17.999 

J18.O00 
-22 ,999 

$24 000 
■29.999 

$30,000 
and over 


4 Year 

1?3 

351 

656 

1,392 

1,774 

2,382 

414 

Pivbl 1C 

All Grants 

1.600 

1.499 

1,388 

1,078 

744 

866 

1,097 


Student Burden 

1J88 

1,418 

1.586 

1,215 

1.355 

1.067 

2.704 

4 Year 

a) 

Expected Family Contribution 

m 

505 

1,109 

1.670 

2,475 

3,976 

51i 

Private 

All Grants 

?,999 

2.829 

2,438 

2,347 

1,944 

1,362 

1.631 


Student Burden 

2,089 

2.341 

2,407 

2,035 

1.887 

1,580 

3.744 

2 Year 

a) 

Expected Family Contribution 

466 

559 

761 

1,126 

1,497 

b7“' 

1,987 

349 

Publ 1C 

All Grants 

1,085 

1,078 

1,033 

792 

590 

585 

978 


Student Burden 

1.086 

1,06) 

926 

985 

785 

918 

2.B48 

2 Year 

a) 

Expected Family Contribution 

598 

459 

607 

1,122 

2,084 

4,162 

409 

Private 

All Grants 

1.654 

1,771 

1 377 

1,138 

1.024 

850 

1,093 


Student Burden 

1,305 

1,282 

1,749 

1,227 

932 

728 

1,976 

Proprietary 

Expected Family Contribution 

471 

554 

1 (106 

1 201 

bl 

1.511 

1 

2.176 

259 


AH Grants 

1,845 

1.789 

1.362 

1,092 

844 

405 

1,342 


Student Burden 

1,958 

1,805 

1.691 

2,151 

2 440 

2,064 

2,733 


Note a) The Expected Family Contribution is computed according to the Unffonn Hethodology, - adjusted where appropriate so 
as not to be less than $1,100 for a student living with parents. 

b) Cell frequency fewer than 30, 

Source* Impact of. the Middle Income Student Assistance Act'll Applied Management Sciences, Silver Spring, MD 

1980. Contract funded by Office of^ Evaluation and Pr<>gram Management /ED. 
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Tables 5 and 6 present only mean student burdens; a fuller picture of the 
affordability of student burdens relative to what students can either earn or 
borrow is presented in table 7. This table shows the results of a comparison of 
individual student burdens with the amounts students could be expected to earn 
from part-time work ($1,350), earn doing part-time work combined with savings 
from summer work ($1,850), or borrow ($2,500).^ The table indicates that all of 
the students attending the lowest cost instutitions ($0-2,500), presumably public 
2-year and 4-year institutions, could have financed their student burdens with 
the subsidized GSL's or NDSL's, and 93 percent could have financed their student 
burdens with work alone. Moving to the next most expensive category of 
institution ($2,501-3,000), 92 percent of the students could have financed their 
student burden with a subsidized loan alone, and 82 percent could have financed 
it by work alone. It is reasonable to assume that an accessible institution near 
every student falls within these first two budget categories, supporting the 
conclusion that access, or more specifically financial access, has nearly been 
achieved. 

Table 7 shows that as institutional costs become greater, the percentages of 
students whose student burdens exceed allowable annual loan limits and 
reasonable levels of part-time earnings increase rapidly. Nonetheless, students 
still attend these institutions, making it reasonable to assume that students who 
are managing to enroll at more expensive institutions could also have attended 
less expensive ones. On the basis of what is presented in this table and in the 
two preceding tables, a considerable degree of real choice among postsecondary 
institutions exists for most students. But for many students choice comes only 
at the price of much higher personal sacrifice in terms of student burdens, 
particularly for the independent student. 

Turning to the question of whether there is equality in the financial terms of 
access and choice among students with different family resources, we can look 
at table 5 again. A comparison of mean student burdens for differing 
dependency and family income categories within institutional cost categories 
indicates equality among dependent students whose family incomes are less than 
$18,000. Mean student burdens tend to decline for dependent students as they 
move to higher income categories and increase markedly for independent 
students. 

Table 8 provides another perspective on how the distribution of grant aid affects 
the degree of equality in the financial terms of access and choice. In this table, 
the hypothetical student burden in the absence of grant aid as a fraction of the 
student's expense budget is compared with the actual student burden as a 
fraction of that same budget. Each cell shows both the mean and standard 
deviation (a measure of dispersion about the mean) of the hypothetical and 
actual fractions for students in the cell. In all cases grant aid greatly reduces 
the percentage of a student's budget that must be financed with loans and/or 
work; the manner in which grants are distributed increases the inequality of the 
percentage for most categories of students. This increase in the inequality of 
student burden as a percent of the expense budget is most pronounced in the 
lowest cost category of institutions ($0-2,500) and in all institutional categories 
for independent students. 

In conclusion, it appears from the data available that ED's student financial aid 
programs have been reasonably successful in providing financial access to some 
level of postsecondary education for all prospective students with financial need. 


37 



Table 7 


Tbe PropOTtion of 600(1 Student Aid Applicants V/hose Student Rurden txcecds $1 350 $1,850 and $2,500 Respectively 


Budget 

Category 

0- 

$2,500 

S.B. Greater Than $1,350 

S.B. Greater Than $1,850 

S.R. Greater Than $2,500 

Dependent Student by Family Incojne 

0 $ 6,000 $12,000 $18,000 $24,000 $30,000 

-5,999 -11.999 -17,999 -23.999 -29,999 and over 

.09 .07 .09 .07 .03 0 

,01 .02 .04 .03 0 0 

a) 

0 0 0 0 0 0 

Independent 

Students 

.29 

.18 

0 

All 

Students 

.15 

.07 

.00 

$2,501- 

S-B. Greater Than $1,350 

.37 

.26 

.24 

.13 

.19 

.04 

.74 

.39 

3,000 

S.B, Greater Than $1 ,860 

.06 

.05 

.04 

,02 

.02 

0 

.35 

.18 








a) 




S.B. Greater Than $2,500 

.02 

.01 

.01 

0 

.02 

0 

.26 

.08 

$3,000- 

S.fl, Greater Than $1,350 

,64 

.66 

.64 

.48 

.60 

.29 

.88 

.69 

4,000 

S.B. Greater Than $1 ,850 

.29 

,28 

.21 

.19 

.20 

.25 

.76 

.42 


S.B. Greater Than $2,500 

.08 

.06 

.03 

.07 

.09 

.17 

,52 

.22 

$4,001- 

S.B, Greater Than $1,350 

.73 

.77 

,72 

.56 

.38 

,48 

.90 

.77 

6.000 

S.B. Greater Than $1,850 

.51 

,58 

.53 

.47 

,27 

.41 

.84 

.67 


S.B. Greater Than $2,500 

.31 

.24 

.21 

.21 

.15 

.16 

.69 

.46 

15,001- 

S.B, Greater Than 11,350 

.91 

.92 

.92 

,81) 

.75 

.58 

.94 

,80 

b ,(J0U 

S.B. Greater Tnan $1 ,odu 

.79 

.G1 

.78 

.08 

.66 

.38 

.91 

.77 


"reater Than $2,500 

.41 

.51 

.54 

.43 

.40 

.17 

,83 

.56 

Over 

S.B, Greater Than $1,350 

.84 

.93 

.92 

.75 

.70 

.55 

.91 

.82 

6,000 

S.fl. Greater Than $1 ,850 

.76 

.81 

.83 

.64 

.65 

.40 

.90 

.75 


S.fl. Greater Than $2,500 

.63 

.65 

.63 

.50 

.47 

.32 

.88 

.66 


*lote' a) Cell frequency fewer than 30 

Source “Study of the Impact of the Middle Income Student Assistance Act'*; Applied Management Sciences i Silver Spring, MO, 
1980, Contract funded by O/fice of F valuation and Program Management /EO 
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Table 8 


The Effect of Grants on the Level and Variation in Student Burden as a Proportion of 'Student 

Budget 1979-«0 


Independent All 

Dependent Student By Family Income Students Students 

l^iidget Student Burden as a 0 $ 6»00O $12»000 $18,000 $24,000 $30,000 

Category Proportion of Budget -5, 999 -11,999 -17,999 -23,999 -29,999 and over 

e) 

H SO H Sn M SO M SD M SD H SO H SD M SO 


0- 

$2,600 

Without Grants 

b) c) 

With Grants 

.89 ,18 

.24 .25 

.86 

.24 

.20 

.30 

.69 

.21 

.22 

.41 

.64 

.09 

.25 .44 

.09 .13 

.27 

.17 

.49 

.18 

.33 

ci) 

.25 

.88 

.48 

.27 

.50 

,82 

' .33 

.25 

.29 

$2,501- 

Without Grants 

.84 .17 

.77 

,22 

.76 

.18 

.63 

.20 .57 

.21 

.45 

.15 

.13 

.89 

,24 

.78 

.23 

3,000 

With Grants 

.34 .22 

.37 

.19 

.37 

.17 

.32 

.16 .32 

.21 

.30 

.63 

.34 

.44 

.25 

$3,001- 

Without Grants 

.91 .12 

.90 

.13 

.80 

.20 

.68 

.21 

.59 

.24 

.55 

.31 

.94 

.16 

.84 

.21 

4,000 

With Grants 

.42 .20 

.44 

.18 

.39 

.17 

.34 

.24 

.36 

.28 

,27 

.38 

.67 

.29 

.50 

.26 

$4,001- 

Without Grants 

.94 ,25 

.93 

.10 

.83 

.19 

.61 

.32 .41 

.33 .51 

.31 

.92 

.20 

.83 

.2? 

5,000 

With Grants 

.45 .10 

.43 

.19 

.42 

.20 

.34 

.27 

.25 

,29 

.20 

.35 

.63 

.25 

.51 

.2? 

$6,001- 

Without Grants 

.93 .14 

.90 

.14 

.84 

,14 

.75 

.21 

.64 

.24 

.43 

.28 

.93 

.18 

.82 

.25 

6.000 

With Grants 

.45 .17 

.47 

.18 

.48 

.19 

.40 

.20 

.35 

.27 

.25 

.24 

.67 

.25 

.50 

.25 

Over 

Without Grants 

.88 .20 

.89 

.20 

.82 

.19 

.68 

.28 .60 

.29 

.40 

.28 

.89 

,26 

.76 

.31 

$G,000 

With Grants 

.44 .24 

.43 

.22 

.44 

.22 

.32 

.20 

.30 

.25 

.21 

.24 

.68 

.28 

.49 

.30 


N os a) Student Burden Without Grants as a Percent of Budget * - fxDected ramily Contribution) - Budget 

b) Student Burden With Grants as a Percent of Budget ** (Budget - Expected Family Contribution) - Budget 

c) The Expected Family Contribution is canputed according to the Uniforw Methodology, adjusted where appropriate 
so as not to be less than $1,100 for a student living with parents. 

d) Cell frequency less than 30 

e) SO - Standard Deviation 

Source: "Study of the twiddle Income Student Assistance Act", Applied Management Sciences, Silver Spring, HD, 1980. 
Contract funded by the Office of Evaluation and Program Management /ED 
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These programs appear to have made even the most expensive postsecondary 
education options potentially affordable even for the poorest students. But 
attendance at high-cost institutions seems to come only at the price of very high 
personal sacrifice for certain categories of students, particularly the 
independent student. 

The Middle Income Student Assistance Act <MISAA) became law on November 1, 
1978. However, its provision affecting the BEOG, SEOG, and CWS programs did 
not take effect until the start of the J979-S0 academic year. The intent of this 
legislation was to reduce the student burden of middle-income students. To see 
if this occurred, one can compare the entries in this r^ort's table 5 with those in 
a similar table in the 1979 Annual Evaluation Report .lP Table 9 presents this 
comparison in terms of the 1978-79 to 1979-80 change in each item. 

What is the net effect of these changes? If we divide the dependent students 
between low and middle income at $12,000 for schools that cost students up to 
$5,000, and at $18,000 for more expensive schools, we find that student burdens 
increased in 12 out of low-income combinations, whereas student burdens 
decreased in 20 out of 22 middle-income combinations. This pattern of changes 
is highly significant statistically.^^ Thus, MISAA appears to have had its 
intended effect with respect to the middle-income students in that the mean 
burden appears to have been more clearly equalized for a number of budget 
groups. To a large extent, this equalization has been accomplished through an 
increase in the mean grant level. However, in a number of these groups there 
was a drop in the mean family contribution as well. Although at first glance, it 
may have appeared that MISAA "robbed Peter to pay Paul," in reality it has 
provided for the equalization of mean student burden levels. 

Student Persistence 


As noted in earlier Annual Evaluation Reports, past studies revealed that 70 
percent of all entering freshmen eventually graduate from college. Fifty 
percent of this group graduated from the institution at which they started} the 
others graduated after transferring to other institutions. Furthermore, 30 
percent of the students who transferred did so in their second year. 

Analysis of recent data collected by an ED study on student persistence indicates 
that financial aid is related to student persistence in a positive manner. In 
particular, those students with a grant or a loan above $1,000 were found more 
likely to persist. Financial aid, as might be expected, assists the low-income 
student more than a student from higher income families. Work aid appears to 
enhance the persistence of low-income students especially if they receive no 
grant aid. Work aid, however, does not appear to be a factor for higher income 
students especially if they already have a grant. 

Improving Access, Choice, and Quality Through Support of Institutions 

The Department of Education also administers programs that grant funds 
directly to the institution. Programs such as these allow the institution to 
maintain and improve the quantity and quality of the educational services it 
provides students without passing the full cost of these services on to students. 
These programs, like the student aid programs, also help the institution reduce or 
hold constant the burden to students in a period of rising costs by reducing the 
net cost of instruction. 
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Table 9 


A Cooiparison of Changes in Dollar Amounts of Family Contributions. -Grant Aid And 
The Student Durden (Work/Uoans) From 1978-79 to 1979-80 by Family Income and By Student Budget a/ 


Independent 

Dependent Students _ Students 


Institution Student 
Expense Budget 


0 

-5.999 

$ 6,000 
-11,999 

$12,000 

-17,999 

$18,000 

-23,999 

$24,000 

-29,999 

$30 000 
and over 



0- 

Expected Family Contribution 

¥ 

32 

72 

"f 

104 

- 

90 

4 

189 

4 

454 

4 

108 

$2,500 

All Grants 

+ 

4 

+ 46 

4 

4 

4 

143 

- 

123 

4 

185 

4 

32 


Student Burden 

+ 

157 

+ 154 

- 

57 

- 

97 

" 

698 

- 

592 

4- 

256 

$2,501 

Expected Family Contribution 

- 

45 

+ 81 

- 

179 

- 

247 

- 

8 

4 

1,088 

4 

84 

$3,000 

All Grants 

- 

49 

- 157 

4 

227 

4 

358 

4 

236 

- 

195 

- 

275 


Student Burden 

- 

81 

+ 62 

- 

52 


414 

- 

225 

- 

1,372 

4 

245 

$3,000 

Expected Family Contribution 

- 

45 

- 137 

- 

31 

- 

112 

- 

377 

- 

263 

- 

4 

$4 ,noo 

All Grants 

+• 

51 

4 116 

4 

297 

4 

513 

+ 

198 

4 

288 

• 

248 


Student Burden 

•I- 

131 

4 144 

* 

97 

- 

281 

- 

112 

- 

442 

4 

213 

$4 ,000 

Expected Family Contribution 

+ 

31 

- 392 

- 

58 

4 

200 

- 

128 

4 

341 

- 

181 

$.5 ,000 

A1 1 Grants 

+ 

113 

4 414 

4 

227 

4 

172 

- 

46 

4 

564 

- 

29 


Student Burden 

+ 

201 

38 

- 

113 

- 

353 

“ 

299 

- 

877 

4 

263 

$5,001 

Expected Family Contribution 

- 

337 

- 262 


159 

- 

117 

- 

646 

4 

285 

- 

242 

$6,000 

All Grants 

4 

145 

+ 21 

4 

28 

4 

324 

4 

221 

4 

156 

4 

112 


Student buraen 

4 

1bb 

+ 37p 

4 


- 

bl 

4 


- 


4 

bU 

Over 

Expected Family Contribution 

- 

188 

- 1,328 


187 

- 

358 

- 

487 

- 

795 

- 

734 

$h,000 

All Grants 

1-1 

,526 

+ 173 


234 

- 

355 

- 

150 

- 

178 

4 

13 


Student Burden 

+ 1 

.013 

4 B34 

4 

905 

- 

19 

4 

76 

- 

52 

+ 1 

,061 


Source' "Study of the Impact of the Middle Income Student Assistance Act", Applied Management Sciences, 

Silver Spring, KO 1980. Contract funded by the Office of Evaluation and Program Management /EO 

Note* a) The purpose of this table is to show how passage of MISAA affected key financial components of student 
resources by family income. No adjustment was made for inflation of family Income and student budgets. 
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Federal institutional aid programs administered by ED make up about 7 percent 
of the total ED higher education budget ($379 rnillion in fiscal yeai 1980 of a 
total of $5,3 billion). ^"2 Title U1 of the Higher Education Act (WE A), the 
Developing Institutions Program, accounts for a little less tharj one-third of this 
total ($110 million in fiscal year 1980). Title III promotes detailed institutional 
program planning, curriculum development, faculty development, and improved 
administrative practices by providing resources to help participating institutions 
finance these programs. Generally, the focus has been on institutions that serve 
large numbers of disadvantaged students. This focus is consistent with the larger 
goal of Federal aid for postsecondary programs; to enhance eciual educational 
opportunity. HEA-Title III program eligibility criteria have included as a key 
factor the relative number of disadvantaged students the institution serves. 

Thus, the program can be said to be on target if the relative number of 
disadvantaged served in developing Institutions exceeds those in the student 
population as a whole. BEOG recipients, the economically disadvantaged 
students, on the average made up 28.5 percent of the student body in HEA-Title 
in institutions compared to 17.9 percent for other 2- and 4-year colleges in 
academic year 1977-78. 


Recent program participants provide the only available evidence on program 
effectiveness. Few participating institutions have closed since the Inception of 
the program. Given that access and choice are Federal goals and since these 
institutions serve large numbers of low-income students, the closing of any 
institution would reduce the number and choice of postsecondary education 
institutions available for students enrolled in the affected institutions,^^ 

Evidence shows that proximity to public and private nonselective colleges 
influences college entrance and that low-income students tend to live at home 
while attending college. The distance from home to college is slronply and 
consistently associated with parental income even after other student 

T? as parental education, ability, and race have been taken into 

1 ® l®sser degree, access are reduced 

v,ith the closing of any institution. 

imoi! assistance programs can be judged in terms of their 

r«a ^ removing or equalizing financial barriers for students other 

postsecondary education Present ond persist in 

no .nore than one in seven students whi mlghj^wile'^ualify.'”* 

sJudenB' e«r“vTf '' P'’‘>8'‘*'".‘<ldntified 

education programs* Currently ^ in postsecondary 

Disadvantaged Students Program is in Special Services for 

report on the impact of the Special Servirec^^’ ^ however, too early to 
5S presented in the program section of volume <^etail on each study 

.niortnation on the Taient Seatch and Educa^“af Sat^crterPr^o^fa^^ 
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In addition, the Department of Education administers a number of small 
categorical and grant aid programs ($30 million or less in annual appropriations). 
Included are fellowship programs, construction assistance, and international 
education. These programs are also discussed in detail in volume II. 


SPECIAL CATEGORY PROGRAMS 
Occupational, Adult, and Career Education 

The major special category programs are authorized by the Vocational Education 
Act (VEA), the Adult Education Act (AEA), and the Career Education Incentive 
Act (CEIA) which, with the exception of the CEIA, are administered by the 
Office of Vocational and Adult Education. The CEIA is administered by the 
Office of Elementary and Secondary Education. 

By far the largest source of Federal support for these special category activities 
is allocated to vocational education programs. The purpose of vocational 
education is to help individuals bridge the gap between education and work by 
providing the knowledge and training needed for specific occupations. Currently, 
funding for Federal vocational education programs is of two basic types; 

o Formula Grants to States 

Allocated on a formula basis, these grants assist States to maintain, 
extend, and improve existing programs of vocational education so 
that persons of ail ages will have ready access to high quality 
vocational training or retraining; to develop new programs of 
vocational education; and to provide part-time employment for 
youths who need the earnings from such employment to continue 
their vocational training on a full-time basis, 

Funds are distributed through three separate authorities; (1) 

Programs for Disadvantaged; (2) Basic Grants to States; and (3) 
Consumer and Homemaking Education. The largest is Basic Grants 
($562,266,000 in FY 1980). 

0 Discretionary Programs 

Three programs avirard grants and/or contracts at the discretion of 
the Secretary: Programs of National Significance (research, 
demonstration, and development projects); the Program for Indian 
Tribes and Organizations; and the Bilingual Vocational Training 
Program. 

Because of a sevenfold rise in public outlays for vocational education during the 
last decade, more than one-third of the Nation's high school students now take 
vocational subjects. While it is true that initial Federal expenditures for 
vocational education programs in 1963 gave impetus to State and local spending 
for these programs, no studies have been conducted to determine whether the 
Federal program’s impact on State/local spending continues. Though it is widely 
believed that Federal dollars no longer stimulate the expenditure of State and 
local funds because of the large State and local overmatch, Federal set-asides 
for national priority groups do have some catalytic effect. For example, set- 


43 


asides for the disadvantaged and handicapped have increased the number of 
programs for these groups. No evidence is available to show whether the 
postsecondary set-aside is responsible for the doubling of postsecondary 
vocational enrollments (e.g., in community colleges) since 1970. 

Measurement problems and interpretation ambiguities make it difficult to 
determine whether or not vocational education has been successful. One 
problem is that employment is not the primary objective of all vocational 
students. Another is that economic conditions probably much more powerfully 
influence employment choice among youth more than does curriculum choice. 
Despite these limitations, the data from four national longitudinal studies 
provide a sufficiently long timeframe to assess employment outcomes for 
vocational education graduates. 


0 Taken together, the longitudinal studies suggest that most secondary 
vocational education have no labor market advantages. The 
exceptions are the young women trained in the office occupations. 
Compared to their male counterparts and other vocational education 
graduates, they experience fewer periods of unemployment, have 
higher hourly wages, and work in jobs related to their training. Other 
high school vocational get jobs not too unlike nonvocational 
graduates within 3 years after graduating; experience similar rates of 
pay and unemployment; have similar degrees of job satisfaction; are 
no more knowledgeable about occupations than nonvocational 
students. Vocational education students are as likely to drop out as 
other students; are less likely than other graduates to continue 
formal education in colleges and universities but will most likely seek 
additional occupational training. Postschool training (outside of 
colleges) pays off more for vocational graduates than for other 
graduates. 

0 Because secondary vocational students have lower aptitude test 
scores and tend to come from poorer- families than other students, 
it is often assumed that these students would lose interest in school if 
vocational education were not available. This hypothesis, however, 
has never been tested. 


Other studies of postsecondary students indicate they are more likely 
to enter training-related jobs. 


The Vocational Education Act supports programs that train persons in specific 
occupational skills and is primarily focused on training at the secondary and 
postsecondary levels. The career education legislation emphasis is on building a 
broad understanding of different types of work at the elementary level with later 
deve^pment of decisionmaking skills in deciding on appropriate occupations. 
m 77 ncentive Act (Public Law 95-207), passed in December 

rilv I formula grants to assist States in implementing career education 

in local schools. It also continues the Secretary's discretionary authority to 
make demonstration grants to State and local education agencies and to 
postsecondary institutions a provision not yet implemented. 


The Adult Education Program focuses on yet another aspect of the transition 
from school to the world of work. Specifically, its legislation provides for 
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support to the States for projects which will assist adults in obtaining the basic 
skills (i.e., language and computation skilis) necessary to function effectively in 
an occupation. 

Programs for the Handicapped 

The several programs authorized under the Education of the Handicapped Act 
(EH A) have one or more of four basic purposes: (1) provision of direct services; 
(2) development, demonstration, and dissemination of new technologies, teaching 
methods, and materials; (3) training regular and special education personnel; and 
(4) program evaluation. In each program, the role of the Federal Government 
has been stimulative, whereby it provides "seed” money to States and other 
grantees to stimulate increased quantity and quality in all services and to assure 
that program benefits reach previously unserved handicapped children. The 
strategy for evaluating programs for the handicapped is to determine if they 
have accomplished their specific purposes and, more generally, have had the 
desired stimulative effect. Accordingly, evaluation studies have been of two 
kinds: (1) those designed to obtain objective data on the impact and 
effectiveness of specified programs, particularly those which represented a 
major Federal investment of funds; and (2) those designed to provide policy- 
relevant planning information to enable the Federal Government to target its 
resources more effectively. 

Studies of the first type found that, in general, these programs have 
accomplished their specific purposes. Efforts to isolate the stimulative effect 
and to demonstrate a causal relationship attributable to Federal programs have 
been complicated by factors outside the control of Federal evaluators and 
program managers. Examples of these factors range from effective lobbying by 
parents and professional groups with special interests in education of the 
handicapped to court cases which have demonstrated that handicapped children 
have not had equal access to educational opportunities. Because of these events, 
an increase in resources for handicapped children has led to a corresponding 
gradual increase in the number of handicapped children receiving services. As 
noted above, the degree to which Federal programs have contributed to this 
increase has not been clearly determined. Furthermore, attempts to 
demonstrate this effect have been complicated by recent legislation which has 
resulted in a significant redefinition of the Federal role in education of the 
handicapped. 

The Education for All Handicapped Children Act of 1975 (P.L. 9(t-142) amended 
EHA, Part B, in the following ways; 

0 It explicity states that Federal assistance to States is to assure 
access to a free, appropriate pubiic education for aU handicapped 
children. 

0 It specifies that the unserved have first priority and that 

the most severely handicapped within each disability category 
who are not receiving an adequate education have second 
priority for services relative to ail handicapped children. 

0 It indicates that handicapped children should be served in the 
least restrictive environment consistent with their needs and 
abilities. 
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It specifies that each child shall have the benefit of an 
individualized educational program which will be updated at 
least annually. 

The Office of Special Education and Rehabilitation Services has data which show 
progress in achieving these legislative objectives. For example: 

o Almost 77 percent of the Nation's handicapped school-age children 
are receiving special education and related services today compared 
to less than 50 percent at the time P.L. 9^-1^2 was enacted. 

0 In the annual child count for the school year 1979-80, it-6 of the 58 

States and Territories (79 percent) reported an increase over 1978-79 
in the number of handicapped children receiving special education 
and related services. 

0 Since the passage of P.L. 9^f-1^2, over 317,000 additional handicapped 
children have been reported to be receiving special education which 
has required the hiring or reassignment of approximately 19,000 
teachers. 


o The number of preschool children ages 3 through 5 

receiving special education has increased by more than 36,000 
in the past U years, a growth of .more than 10 percent. 

0 Since the enactment of P.L. 9<^-R2 in 1975, the number of 

previously institutionalized handicapped children being served 
by local districts has increased by almost 61 percent. 

Developmental Programs in Education 


A rnajor purpose of several programs administered by t 
quality of educational practices. These programs are; 


Basic Skills Improvement 
Community Education 
Consumer Education 
Education in the Arts 
Gifted and Talented Education 


Indian Education, Part B 
Indian Education, Part C 
Metric Education 
Teacher Corps 
Teacher Centers 


Community Educatfon! TekS 

Improvement. Right-To-Read portion of Basic Skills 


Cornmunity Education 

IomS°rXa5to„7n'drs?L'r =«ec.ive practices in 

recent study indicates that the proRrmtTha^ 'hese practices. A 

purposes. For example: ^ ® ^ effective in achieving its 

° P-Pira“^d‘ ‘"?ap“J.:tL'l^r';eT^ -PP-*- 
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o These levels of program effort and activity were achieved, 
in part, because there were substantial commitments of 
non-Federal resources supplementing the Federal funds. 

o The majority of the projects which had received Federal 
funds for only one year were continuing at least some of 
these activities and in several, the level of effort 
even without Federal funds liad increased. 

o These project-supported activities had produced desired 
effects on the educational systems and broader community 
and also produced identifiable changes in individuals which 
were appreciated by the project participants. 

The primary negative finding was that recordkeeping at the project level was 
inadequate. Routine, programmatically meaningful data do not now exist. 

Teacher Corps Training 

Teacher Corps is currently implementing a new program whereby its field 
projects in partnership with an institute of higher education (IHE), a school 
district, and a community advisory group, provide inservice training for teachers 
that is responsive to both district and community needs, with a hoped-for side 
benefit of reforming IHE teaching practices. Additionally, the training is 
intended to instill a greater sensitivity to the needs of minority and/or 
disadvantaged children so that the teacher can work more effectively with these 
children. Teacher Corps expects to demonstrate several program models 
directed toward these objectives. 

A study of Teacher Corps graduates of preservice training projects showed 
partial success in achieving similar objectives. Teacher Corps graduates were 
most different from control group teachers in (1) developing ethnically relevant 
curricula; (2) using community resources in teaching and for contacting parents; 
and (3) having positive attitudes about reading development and causes of 
poverty in the society. 

These findings reflect Teacher Corps' special concern about low-income minority 
group children. However, teacher performance measures in the study showed no 
differences in such categories as teachers being a change agent in the school or 
in interaction between teacher and pupils in the classroom. Further, no 
significant differences between Teacher Corps and control group classes were 
revealed on any reading measure, despite a greater emphasis on reading 
instruction and academic subjects by control group teachers in grades 2-3. 
Teacher Corps graduates, however, brought about significantly greater changes 
in a child's self-concept than those brought about by control group teachers. 
These changes consisted of observed expressions of greater happiness and greater 
self-worth in the classroom, and better scores on important subscales of the 
Piers-Harris self-concept scale. 

Basic Skills Improvement — Right-To-Read 

This new program extends the scope of the Right-To-Read program, with 
increased responsibilities for improving written and oral communication skills, 
and mathematics and reading skills. The transition from Right-To-Read to Basic 
Skills Improvement took place in October 1979. 
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Case studies of the Right-To-Read State Leadership Program indicate that the 
State components of the Right-To-Read program^ with a modest amount of 
money, appears to have been successful. For example; 

0 State Right-to-Read programs have made considerable efforts 
to involve large numbers of local districts in Right-To- 
Read efforts. 

o Training has been a major focus in the State Right-To-Read 

Programs. Local district directors received from at least 30 to 
more than 200 hours of training. 

o Data indicated that States wiii continue Right-To-Read even in the 
absence of Federal funds. 

o While Right-To-Read programs have been implemented at 
least to a minimal degree across grade levels, the major 
programmatic focus of Right-To-Read has been at the elementary 
level. 

o Reading has become a top priority in State and local education 
agencies. 

None of these outcomes relate directly to the reading achievement of students, 
because the study was designed solely to assess type and level of technical 
assistance activities (none of which have a direct effect on students) fostered by 
the State component of Right-To-Read funding. 

Overcoming Obstacles to Equal Education Opportunity 

Several ED programs are authorized to help overcome obstacles to learning. 
Among those with recent data are programs for Native Americans and one for 
preventive education on alcohol and drug abuse, 

Indian Education 

Projects to meet the special education needs of Indian children are funded under 
Part A of the Indian Education Act. A study of this program found that: 62 
percent of these projects focussed on instruction in cultural heritage and native 
language; 58 percent emphasized remedial reading; 56 percent emphasized self- 
concept objectives; and 46 percent emphasized remedial mathematics objectives. 
Innovative projects were not always effectively implemented. However, the 
study shows that effective implementation appears to be a function of the 
project’s grant size, objectives, parent involvement, and Indian population 
density. 

Alcohol and Drug Abuse Education 

This program has implemented a new strategy which it describes as "teams 
teaching teams." It is hoped that the new strategy will achieve a multiplier 
effect through a two-step process: (1) teams are trained in "clusters" rather than 
in geographically unrelated groups; A ^ (2) the most impressive trainees in these 
clusters become, through further training and i year of onsite technical 
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assistance, new trainers for other schools and groups of schools in their district 
or community, while continuing to receive some technical assistance from the 
Federal program. 

Two studies funded by the program found that: (1) trainees felt their training 
was effective^ (2) one-half to two- thirds of the teams were still functioning a 
year or two after training? (3) the majority (about 80 percent) of the teams were 
performing activities for which they had been trained; and (^) in the judgment of 
the evaluators, these activities were effective. There has been no definitive 
evaluation of impact on student behavior. However, voluntary reports from 
operating teams cite evidence of substantial decreases in suspensions, vandalism, 
and discipline infractions. Several teams have provided evidence of a multiplier 
effect both by increasing the numbers of teams (and therefore schools covered) 
through training and, in some instances, by raising financial support. Since the 
program does not provide financial support for such reporting, many teams do 
not provide reports and, therefore, it is not claimed that the reporting teams are 
representative of all teams trained. 

Learning Resources To Improve the Educational Experience 

A. number of programs administered by ED do not contribute directly to the 
educational process but do develop resources which serve as important adjuncts 
to the process. The primary function of most programs in this category is to 
improve library services; three other programs are involved in some aspect of 
educational television. The programs are; 

Libraries and Learning Resources 

Library Services 

Interlibrary Cooperation 

College Library Resources 

Library Career Training 

Library Research and Demonstration 

Strengthening Research Library Resources 

Educational Television and Radio Programming Support 

Emergency School Aid Act (ESAA)- TV Programming Support 

Library programs provide a variety of support ranging from grants for 
"hardware" (e.g., equipment) and "software" (e.g., recorded instructional 
packages) to upgrade the resources of school, college, and public libraries, to 
training fellowships, and to research and demonstration projects to improve the 
quality of library practices and services. The programs focused on educational 
television and radio either support program (series) development or improvement 
of broadcast facilities. 

ESAA-TV 

As part of an ongoing study of this program, a survey was conducted to 
determine the extent of viewing of TV series produced with ESAA support. The 
survey findings include: 

o Availability to Potential Target Audiences. For the 11 TV 

series available for viewing, the percentage of potential target 

audiences residing in areas where the series were aired 

ranged from 3 percent to 81 percent. Four series could be viewed by 
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between 32 percent and <f2 percer>t of the target group, and four 
other series could be viewed by 74 percent to 81 percent of the 
potential target audiences. 

o Awareness of the TV Series. A relatively small proportion of children 
were found to be aware of the TV series. The percentages of 
awareness generally fell in the 20 percent to 40 percent range but 
went as low as 4 percent and as high as 55 percent. Overall, less than 
a third of the children had heard of the ESAA series being shown in 
their areas. 

o Viewing at Some Time in the Past. Generally, between 10 and 25 

percent of the children reported having ever seen an ESAA-TV series. 

o Viewing Last Week. The proportions on this item were low, with 
most ranging between 0 and 5 percent. 

o Preference of ESAA Series. Children were asked to indicate 

their preference between the ESAA series and its competitors in 
the same time slot. In grades 1 and 4, between 5 and 
10 percent indicated preference for the ESAA series? in 
grades 7 and 10, between 0 and 5 percent indicated 
such a preference. 

The viewing data can be interpreted in three ways. The first compares ESAA-TV 
to entertainment television. Given this standard of comparison, the low 
viewership of the ESAA-TV series may be a cause for concern, A second 
standard of comparison sees ESAA-TV as educational/public television. The 
viewership findings indicate that ESAA-TV is comparable to other public TV 
programs, which always draw a smaller share of the audience than entertainment 
programs. Finally, ESAA-TV can be perceived as an alternative means to reduce 
cultural isolation. This perspective seeks to examine the number of children 
served through various alternative mechanisms? however, the necessary data for 
this comparison are not presently available. 

Evaluative studies have recently been completed for the Library Research and 
Demonstration Program and the Interiibrary Cooperation Program. 

The Library Research and Demonstration Program (Title II-B of the Higher 
Education Act) has had a significant impact upon the library and information 
community. Title II-B provided millions of dollars for innovative research and 
demonstration? previously few funds were available. This is the only Federal 
program that supports a wide range of research to improve public library 
services. It has generated new knowledge about innovative approaches to 
networking, serving special target groups, using community resources, and new 
technologies. Several projects have been successful locally. A few have had 
national Impact, such as the series of projects which contributed to the 
development and improvement of the Ohio College Library Center, 

Program weaknesses include lack of adequate national dissemination, failure to 
adopt successful practices, and lack of a national program strategy. 

The Interiibrary Cooperation Program, Title III of the Library Services and 
Construction Act (LSCA), has influenced the development and expansion of 
library cooperation in a number of ways. It is a major force behind the 
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development of multi-library cooperation and networking, primarily at the State 
ievel. Along with Title I of this Act (Library Services Program), Title III has 
resulted in greater centralization of planning and administering library services 
at State and regional levels. It was also credited as a major influence on State 
legislatures to modify or pass legislation favoring cooperation and networking. 
Activities and services to increase access to library resources and to provide 
librarians with needed continuing education were listed as the major outcomes of 
the cooperative and networking projects. 

Although Title 11! has had a definite, positive impact on the development of 
multi-library cooperation and netvvorking, sev'atal shortcomings and anomalies 
were identified. I-irst, signific.-pt a'iiO’inf> of LSCA Title I funds (approximately 
20 percent of t'p* fiP'Ml yt-ar I07h apprapi lation) were used by the States to 
partially oi eo'njd'-'triv support roultilibrary -'•ooperation and networking 
activitio'j. It 11 . nut v,'h“ihi‘i this us‘* is detrimental to the mandate of 

Title I (developsiicnt and '.uppin of States efforts to expand and extend public 
library sen.u'f’s to tin* Ufv.ervt d and under->.-;rved), or if this is an effective means 
of supporting tli'-'M* types of pruje-ts. 

Second, LSCA 1 itle 111 was providing support for projects that had become 
integrated into the States' basic programs of library services. This finding should 
not be interpreted as indicating tfiat tiie States are misusing Title III funds. The 
legislation clearly specified thut Title 111 funds can be used to support 
cooperatives and networks. However, given the modest funding level of Title 111 
and the level of use of LSCA Title I and State funds, a shift in emphasis may be 
considered. Slightly more than 8^ percent of the projects receiving Title III 
funds were ongoing projects, scheduled to continue operations for an indefinite 
period of time. Only 14 percent of the projects were short-term, trial projects. 
These figures suggest that Title 111 funds are targeted more at operational 
activities than at riskier research and development activities. 

Further, the range of cooperative and networking services supported by Title III 
projects was restricted. The interlibrary loan and reference and referral 
services were probably efficient and a proper means of gaining access to a wide 
variety of libraries. But there was some question as to how many patrons 
actually needed and used these services; and there were a number of other 
essential library support services (e.g., document access and sharing, 
warehousing and shared storage of low-use materials) which were not being 
provided. The question raised is; How niore Title III funds can be directed 
toward exploring new forms of cooperation and networking which will further 
increase the efficiency and economy of library services, as well as increase 
access to library resources. 
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NOTES 


^Of the 5 million children who scored in the bottom quarter, only 32 
percent (1,6 million) participated in Title I. 

^For these and other reasons, State allocations of Title I migrant funds are 
less than what they might be if all migrant children were identified properly. 

3 Fully funded in this sense means that allocations to States are a product 
of the number of eligible children and the authorized per pupil expenditure, and 
the grants are computed and funded "off-the-topj" that is, even when the Title I 
total appropriation falls short of the authorized level, grants in these two 
programs remain undiminished. Grants to school districts under Part A are 
reduced as necessary. 

A recent review of educational evaluation recommended greater 
attention to measuring implementation. See Robert F. Boruch and David 5. 
Cordray, A Comprehensive Study of Evaluation Practices and Procedures in 
Federally Funded Elementary and Secondary Education Programs. Study funded 
by the U.S. Office of Education, 1980. 

^College enrollment rates over time for various subpopulations (defined by 
characteristics such as family income, sex, race, and ethnicity) indicate changes 
in students' pursuit of postsecondary school opportunities. However, enrollment 
rates by themselves say nothing about how overall opportunities for college-age 
youth have been changing. For example, any potential Increase in enrollment 
figures for low-income or minority students by increasing available financial aid 
may be offset entirely or partially by improvements in employment prospects for 
noncoilege-educated individuals. Such a phenomenon would be most noticeable 
for students whose performance in college is predicted to be "marginal.” Tables 
presented in this section should be interpreted with this in mind, 

%he $1.9 billion is the Federal subsidy for interest, defaults, 
administrative costs, and capital contributions for these programs. However, the 
actual value of loans made under the program totaled to $3.5 billion. 

T^The State Student Incentive Grants (SSIG) Program is not included in this 
discussion, since it provides matching grants to States to be used in their own 
student-aid programs and, as such, cannot be differentiated from State funds by 
recipients or financial aid administrators. 

^It should also be remembered that there are many students above the 
mean. Thus, the mean is only a broad measure of impact and should be thought 
of as such. 

^Each cell entry indicates the precentage of aid applicants in the cell 
whose student burden exceeds the amount indicated at the left — $1,350, $1,850, 
and $2,500. 

^^See Annual Evaluation Report on Programs Administered by the U.S. 
Office of Education; Fiscal Year 1979 , U.S. Office of Education, HEW, p. 256. 

insignificant at less than .01 using a chi-square test. 
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^^This amount includes funds for the Special Services Program for 
Disadvantaged Students and State Grant Programs. 

^^This statement should be qualified in that the students may find other 
opportunities of equal or higher quality. 

^^Higher Education Research Institute, The Impact of Student Financial 
Aid Programs on Student Choice , Draft Final Report, 197S. 

^^Local and State studies are flawed because of insufficient followup and 
inability to control local economic fluctuations and mobility of graduates. 

16^ geographically related cluster is comprised of a team from a high 
school joined with teams from the elementary, middle, and/or Junior high schools 
feeding into that high school. 
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Chapter ill 


Recent Evaluation Results; 
Highlights and Uses 




This chapter highlights important findings from recent evaluation studies an 
describes the specific uses of evaluation results. It outlines in broad perspective 
the use of results from ED evaluation studies. Detailed information is included 
in the individual program descriptions in volume II. 


For each major study, the Office of Evaluation and Program Management 
prepares a 6- to 10-page Executive Summary which briefly describes the 
background, methodology, and findings of the study. Persons interested in 
greater detail about a study than can be presented in this chapter should call or 
write for a copy of the Executive Summary, as follows: 


Ms. Yvonne Briscoe 

Office of Evaluation and Program Management 

Room t^Q70, Switzer Building 

U.S. Department of Education 

Washington, DC 20202 

<202)2^5-0361 


Evaluations attest to program successes and failures, strengths and weaknesses, 
and thus provide the primary source of objective evidence used in determining 
future program operation and policy options. To be useful, however, evaluation 
and management study findings must be available when needed by Department 
officials for budgets, planning, and operational program decisions and by the 
Congress for appropriations and reauthorization decisions. As outlined in 
chapter 1, the Office of Evaluation and Program Management has concentrated 
on evaluation followup to promote findings that assist in important 
decisionmaking. Executive Summaries, staff briefings, the Program Resources 
Center, and Evaluation Implementation Boards are all mechanisms for 
incorporating evaluation findings into the decisionmaking process. 

The program evaluation findings described in this chapter can be categorized in 
four areas, based on their use for policy officials and program managers: 
legislation; program effectiveness; program management; and program content. 

Study of Evaluation Practices and Procedures (Legislation) 

The recently completed "Study of Evaluation Practices and Procedures" by 
Boruch and Cordray of Northwestern University recommends that policymakers 
in both the executive and legislative branches take a variety of measures in 
mandating and conducting evaluations. In response to the "Holtzman mandate" 
in the Education Amendments of ‘1978 (Section 1526 of Public Law 95-561), the 
report recommends that the Congress: 

o Clarify statutory provisions for evaluation and participate in regular, 
joint planning for evaluations with the Department of Education. 

o Conduct a "capability assessment" of grantees' abilities before 
requiring new evaluation tasks. 

o Provide for pilot tests of new program variations in which high 

quality evaluation designs such as randomized studies can be used. 

o Encourage by law independent analysis and critique of evaluations. 
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o Refer more often to evaluations used in deliberations so that uses of 
evaiuations can more easily be tracked. 

o Refrain from incorporating specific evaluation guidelines into law. 

Similar recommendations were made to the Department of Education. The 
report further advised Federal evaluators to consult with congressional offices 
more regularly, make evaluation reports more quickly and easily available, and 
include more systematic measurement of program implementation in evaluation 
reports. 

Program Consolidation and the State Role of ESEA-Title IV (Legislation) 

A recent study of the ESEA-Title IV consolidation effort indicated that most 
States and local school districts manage Parts B and C of Title IV independently, 
as they did for the categorical programs subsumed by Title IV, In Title IV, the 
Congress attempted to consolidate programs that under previous legislation and- 
organization were not procedurally or substantively compatible. The absence of 
administrative consolidation at the Federal level apparently influenced States in 
their administrative decisions. A related lesson is that just as the Federal 
response must represent substantive change, so must a consolidated policy 
represent more than a shuffling together of categorical legislation. 

Furthermore, the Title IV legislation in the Education Amendments of 1978 
contained no new language to suggest congressional commitment to a 
consolidated program effort, nor did it include incentives to encourage the 
institutional changes assumed by consolidation. 

Validation of the Student Counts Used To Allocate Funds in Migrant Education 
(Prograrii^ Managemeint) 

information from a recently completed study of the 1977 counts used to allocate 
fiscal year 1979 funds for the Migrant Education Program indicated a 12 percent 
undercount of full-time equivalent (FTE) students enrolled in the Migrant 
Student Record Transfer System (MSRTS) during 1977. The major reason for the 
migrant children undercount appeared to be a failure by school districts to 
submit complete enrollment data to the MSRTS. Roughly 15 to 20 percent of the 
eligible migrant children enrolled in school during early 1978 were not enrolled in 
the MSRTS in 1977 and therefore did not accrue FTE credit. About 2 percent of 
the FTE's were generated by ineligible children; another 2 percent of the 1977 
FTE's were generated by eligible children while they were out of the country. 
Overall, the 1977 counts (both'for the Nation as a whole and for the geographic 
regions defined for the study) seemed to represent a conservative estimate of 
the total number of migrant children. 

Evaluati on of ESEA Title 1 Programs for the Neglected and Delinquent ( Program 
Management) ° 

The recently completed National Evaluation of State Programs for 
Institutionalized Neglected or Delinquent Youth under Title I of the Elementry 
and Secondary Education Act describes the program in terms of student need for 
services and actual benefits from participation in Title I, Overall, there is 
evidence of significant need but of irregular service delivery. In addition, there 
is scant evidence of positive change in either basic skills achievement or student 
self-perception. 
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On a positive note, special efforts to define characteristics of successful 
programs, as well as practices which impede student progress, resulted in 
information which local and State administrators can use to improve their 
programs. For example, a modest relationship was found between student 
exposure to basic skills instruction and achievement growth; yet over half of the 
class-time scheduled for basic skills instruction was actually used for 
noninstructional activities. Of this amount, more than half resulted from 
student absences often due to assignment by the institution to other activities. 
Only about 25 percent of the class time which was non-task-related could not be 
controlled by staff; the rest of the time went to activities such as smoking 
breaks, clean-up, tracking missing students, and casual conversations. Another 
practice to avoid was the misuse of audiovisual materials to free teachers from 
teaching. 

In settings where teachers or aides provided instruction directly, reading gains 
were greater; the fewer students per teacher and the less time teachers and 

students spent in non-task-related conversations, the greater were math and 
reading gains. 


Characteristics and Contexts of ESAA Basic Human Relations Programs 
(Program Content) 

A recent study of ESAA-funded human relations programs reported an increase 
in the number of these programs over the last few years. The study also 
reported that; 

o Compared to all ESAA districts, those with human relations programs 
tended to be larger, more recently desegregated, and to have a lower 
percentage of minority students in the school population. 

o Two types of human relations services were offered to students: 
services focusing primarily on intergroup relations, problems, and 
understanding; and services focusing primarily on the special needs of 
minority students. 

o Programs with a special needs focus occurred in districts that 

desegregated early, while more recently desegregated districts were 
likely to have inter group relations programs. 

o Human relations services were also offered to staff and parents. 

Staff services consisted primarily of Inservice training sessions, and 
participation in advisory groups was the most common form of 
service to parents. 

Later phases of this study will examine the effectiveness of these services and 
the characteristics of successful human relations programs. 

ESEA, Title I Early Childhood Education (Program Content) 

A recently released report of an ongoing study of early childhood compensatory 
education programs concluded that early childhood Title 1 programs, both within 
and across grades, have extremely diverse strategies and objectives. All the 
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general educational trends common in early childhood education are evident in 
Title 1 projects. The report also observed that: 

o Prekindergarten programs typically address a wide range of goals— 
cognitive, language, emotional, social, gross-motor, psychomotor, 
organizational, and motivational development. 

o The structure of kindergarten Title I programs depends on the 
availablility of other projects within the school district. 

o Virtually ail Title I first-grade programs are child-focused, school- 
based, and part of a multigrade-level program in reading, language 
arts, or mathematics. 

o Without standards or guidelines on how to conduct required 

evaluations of early childhood programs, districts use many methods 
of varying quality. 

o Methods to evaluate parent involvement and the consequences for 
their children need to be developed. 

The findings of this report are being used to review alternative strategies for an 
evaluation system for early childhood Title 1 programs. 

State Student Incentive Grant Program (Program Management/Content) 

This study investigated issues concerning the relationship between State and 
Federal agencies in administering student financial aid, with emphasis on the 
State Student Incentive Grant (SSIG) Program. SSIG is a $76.5 million per-year 
program designed to encourage States to develop or expand programs of grant 
assistance to needy undergraduate students. Currently, it is a 50-50 cost-sharing 
(State-Federal) program. 

The States were found to be sharply divided in terms of the amount of funds 
provided for student aid. Sixteen States exactly match SSIG funds with State 
aid, while 13 States contribute more than 10 times the SSIG dollars received. 
Further, the needs analysis procedure used to determine student eligibility for 
aid, the average and maximum awards, and the administrative structure 
responsible for State grant programs, vary greatly from State to State. 

The study also found that, at almost all institutions, the tuition refund policy is 
described in the school's admissions catalog. Twenty-five percent notify 
students of refund policies prior to registration, and another 25 percent give 
notification to students upon registration. The average interval between 
application for and receipt of a refund is 2Vt weeks. Finally, 66 percent of the 
public institutions and 60 percent of the private institutions have established 
procedures for students to appeal unfavorable refund decisions. 

Criteria for Recognition of Accrediting and State Approval Agencies (Program 
Management) 

Recognition by the Secretary of Education of accrediting and State approval 
agencies is the principal means of establishing eligibility to participate in 
Federal student aid programs by postsecondary education institutions. An 
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evaluation of the criteria and procedures for recognition was recently 
completed. Highlights of the study findings are: 

0 The ED procedures for recognition reliably differentiate ineffective 
agencies from effective ones, and these procedures are consistent 
over time in interpreting and applying the criteria. 

o The recognition decisions over the 10-year period from 1969 to 1978 
have consistently utilized virtually all the established criteria. 

o Agencies denied recognition differ among themselves more widely 
than do the agencies recognized. The major dehciencies of the 
agencies denied recognition involve weak evaluative procedures, 
questions about the impartiality of their decisions, and 
nonrepresentative governing boards. Other deficiencies are limited 
staff, budget, and experience. 

0 When presented descriptions of agencies that had been evaluated and 
either recognized or denied recognition by ED, a group of slightly 
more than 100 persons knowledgeable about accrediting agreed 
substantially with the ED's decisions. The measure of agreement 
between the actual decisions and the judges' decisions was .68, where 
1.00 indicates perfect agreement. Agreement was nearly unanimous 
for agencies denied recognition. The major source of disagreement 
involved agencies recognized for a limited period, where judges 
tended to award recognition for a slightly longer period than actually 
took place. 

A Guide To Measuring Achievement Level and Program Impact on Achievement 
In Projects (Program Management/Effectiveness) 

One outcome of a study entitled "An Evaluation of Project Information Packages 
(pip's) As Used for the Diffusion of Bilingual Projects" was the preparation of a 
prototype evaluation manual. This manual contains specific achievement- 
evaluation guides and worksheets which were developed for use in this study. 

The manual is intended for local bilingual education project directors and 
evaluators who must work together to plan and implement local evaluations. 
While it is not a complete guide to bilingual program evaluation, it focuses on 
two important questions; 

o What is the level of student performance relative to national 
norms or comparable groups? 

o What is the impact of the bilingual program on student achieve- 
ment compared to results of other local instructional practices, past 
and present? 

This unconventional guide to evaluating bilingual projects deals in-depth with 
selected problems that are either unresolved or widely overlooked in current 
evaluations, and it recommends solutions for evaluators at the local level. 

Title I Technical Assistance Centers (Program Effectiveness) 

The Technical Assistance Centers for Title I of the Elementary and Secondary 
Education Act provide evaluation technical assistance for Title I elementary and 
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secondary programs to State education agencies and, through the State agencies, 
to iocal education agencies. There are iO Technical Assistance Centers (TAC's), 
one for each ED region. Many of the problems faced by the State and local 
education agencies across the country are the same (e.g., test selection, quality 
control, implementation of the Title I evaluation models, measuring whether 
gains are sustained, and use of the test and evaluation information). 

To avoid duplication of effort, development of materials was coordinated across 
the TAC's. Thus, materials produced by one TAC are available to all other 
TAC's and to State and local agencies as well, and TAC's are required, whenever 
possible, to use existing material. 

Bilingual Student Placement System (Program Management) 

Developed under contract in 1980 by ED, the Student Placement System offers 
resources to assist bilingual education projects funded under Title VII of the 
Elementary and Secondary Education Act in the design and implementation of a 
system for; (1) selecting those students most in need of project participation; (2) 
deciding when a participating student may successfully be transferred to an 
English-language classroom; and (3) providing appropriate followup assistance to 
transferred students. In summer 1980, copies of the student placement system 
were distributed to ail Title Vll-funded bilingual education projects. In fall 1980, 
a series of training sessions was held to train staff of the Bilingual Education 
Service Centers and staff of Evaluation, Dissemination, and Assessment Centers 
and, subsequently, to train staff of local projects how to use the Student 
Placement System. 

Prograni Management Procedures in Campus-Based and Basic Grant Programs 
(Program Management) 

The rapid expansion of Federal student financial aid programs has put great 
strain on a management "system" that can at best be described as "varying." A 
recently completed study identified significant gaps that persist in the quality of 
management practices at the institutional level. Four management procedures, 
in particular, were identified as needing improved administration: 

o Clarifying and enhancing the role of the student financial aid office 
in the institutional hierarchy. 

o Developing and implementing a student financial aid information 
system, at both the precollege and college level. 

o Expanding financial aid counseling for special student populations, 

o Clarifying and refining the packaging of student financial aid. 

In addition to identifying problems, the study discusses related issues in detail 
and offers recommendations to help improve institutional operations. The study 
does not attempt to duplicate existing manuals on the subject. It does, however, 
review existing publications on the subject for the reader. It also contains 
program impact information. 
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Impact of the Middle Income Student Assistance Act (Financial Aid) 


A study of MISAA's impact showed that high proportions of dependent students 
from all family income levels were receiving aid, but that the proportionate 
increase was greater among those from families with incomes in excess of 
$12,000. As a result, the percent of college costs met from work or loans for 
students with family incomes in excess of $12,000 declined significantly, 
regardless of type of institution attended. Family contribution expectations also 
declined, according to a recent study of Federal student aid programs. 

At the same time, little change in the relative distribution of need-based Federal 
student aid awards among ^-year institutions was observed. Two-year colleges 
gained as a group while proprietary institutions, which serve large numbers of 
low income students, showed a loss. All institutions reported increases in the 
average student award ranging from $100 to $300. 

Also reported were the concerns of financial aid officers regarding regulations, 
administrative costs, timing of funding, and reporting requirements. 

Upward Bound (Program Effectiveness) 

A recent study of the Upward Bound (UB) program indicated that: 

o About 91 percent of the typical UB participants entered some type of 
postsecondary education while about 70 percent of comparable 
nonparticipants entered. 

0 Of the students who entered postsecondary education, about 73 
percent of typical UB participants attended a 4-year college or 
university. The comparable rate for nonparticipants was 50 percent. 
About 22 percent of the nonparticipants attended a 
vocational/technical school in comparison to 9 percent of the UB 
participants. 

o Typical UB participants received more financial aid to attend 

postsecondary education than comparable nonparticipants. The UB 
participants received twice as much grant/scholarship funding (i.e., 
$1,428 vs. $689 on average) as comparable nonparticipants. Typical 
UB participants also received larger loans (i.e., $393 vs. $300) than 
comparable nonparticipants. 

o Overall, UB had a large positive effect on student persistence. 

Across all types of schools, typical UB participants maintain their 
enrollment about one term longer than comparable nonparticipants. 
After the type of school attended is taken into account, UB 
participants persist slightly longer (i.e., one-fifth of a term) than 
comparable nonparticipants. 

o The grades earned at postsecondary institutions by UB participants 
were roughly the same, or slightly lower, than those earned by 
comparable nonparticipants. At 4-year colleges and universities, 
average UB participants and comparable nonparticipants earned 
grade-point averages of about 2.0 (equivalent to a "C"). 
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Financial Indebtedness and Postsecondary Education (Program Content) 

Preliminary findings from a study of student indebtedness indicate that the 
median student debt at the bacculaureate level increases from $3,000 to $10,000 
for students attending graduate school. Highest as a group were medical 
students who entered with a median loan amount of $5,000 and saw it grow to 
$15,000, while at the low end of the scale the median indebtedness of education 
majors grew to $3,000 from a base of $2,000. The study also compares expected 
earnings, by discipline, with the level of student indebtedness. 

Sex Equity in Vocational Education Programs (Program Effectiveness) 

This study was mandated by the Congress to determine to what extent sex 
discrimination and stereotyping in vocational programs had been reduced or 
eliminated. Findings indicate that overt discrimination, such as rules denying 
admission on the basis of sex, has been reduced. However, many practices still 
discourage students from entering nontraditional fields before they are enrolled 
and further discourage them once they are enrolled. Despite societal factors 
outside the school which exert powerful influences on the aspirations of youth, 
study findings indicate that the school can have an impact because of the strong 
positive correlation between the school's level of equity activity and its 
proportion of nontraditional enrollment. However, little action was evident at 
the school level to foster equity —particularly student-oriented activity. 

Few States or schools were engaged in community and employer activities. Yet 
these activities — particularly with employers — seem to be a critical ingredient 
in any successful effort to promote sex equity. There is also evidence that 
efforts to achieve equity in staffing patterns have a positive effect. Those 
schools with lugher "internal affirmative action" indexes also had higher 
proportions of nontraditional teachers. 

National Study of Vocational Education System and Facilities (Program Content) 

A comprehensive study of the vocational education systems in each of the States 
and Territories has provided the first thorough description of State governance 
structures, delivery systems, and funding arrangements. Prior to this study, 
heterogeneity among the States was known to be characteristic of vocational 
educations the study documents these differences. During the course of the 
study, information gaps recognized at the outset were found to be even greater 
than anticipated. 

The report deals with the organization, governance, and financing of the State 
systems; the organization at the local levels within each State; and State 
financial policies as they relate to vocational education facilities, equipment, 
and program operations. The number, characteristics, and utilization of the 
several types of vocational educational institutions are also assessed for the first 
time. 

The general characteristics of 6,660 institutions (5,560 secondary and 1,100 
postsecondary) are described. Among these characteristics are the number and 
geographical distribution of the institutions, shops, and laboratories; types of 
construction of facilities; reported condition of facilities; and accessibility to 
handicapped persons. 
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other findings sl^ow tluit: 


o The greatest concentration of vocational education facilities (shops 
and levboratory spaces) is in communities of 25,000 - 100,000 
population. 

o In 1965, 40 States financed construction projects with Federal 

vocational education funds; in 1975, only 21 States used these funds. 

o The proportion of vocational education facilities reported as totally 
inaccessible to the handicapped is greatest among vocational high 
schools: over 30 percent are inaccessible, compared to lesser rates 
at other other types of vocational education institutions. 

Assessment of State Plans for Career Education (Program Management) 

A recently completed study has provided 49 States and Territories with 
evaluations of their State plans for implementing career education. Initial 
support for developing these plans was provided during fiscal years 1976 and 
1977. The study, which started in September 1978, identified 10 elements of a 
model State plan. Criteria were developed so that each element of a plan could 
be rated on a three-point scale as being "strong," "adequate," or "weak." An 
overall rating was then calculated on a five-point scale. 

The 49 States and Territories which submitted plans before May 1979 were given 
preliminary ratings of their plans. In these ratings, 23 (47 percent) received an 
overall rating of 3.0 or higher. In response, over half of the sites (57 percent) 
voluntarily provided revisions and/or additions to remedy reported weaknesses. 
Final ratings were then sent to all States. This time, 30 (62 percent) were rated 
3.0 ("adequate") or better. Seven of the plans were rated "strong" and only three 
were considered "weak." 

The States and Territories also received a guide to help them adapt their plans to 
meet the requirements of P.L. 95-507, the new Career Education Incentive Act. 
This guide together with the assessment under the old law enabled States to 
incorporate the information gained from the individual evaluations of their plans 
into the new 5-year plan required by the Career Education Incentive Act. These 
plans form the basis for funding in fiscal year 1980 and beyond. 

A Study of Library Cooperatives, Networks, and Demonstration Projects 
(ProRram Effectiveness) ^ 

A study of the Library Research and Demonstration Program and the Interlibrary 
Cooperation Program found that both programs were making significant 
contributions to the improvement of libraries. 

The Research and Demonstration Program has provided millions of dollars for 
innovative research and demonstration, where either very few or no other funds 
were available. Local projects often focused on providing library service to 
minorities and on the utilization of new technologies. 

The Interlibrary Cooperation Program is responsible in part for the development 
of multilibrary cooperation and networking, especially at State and regional 
levels. It was also found to be a major influence toward modifying or passing 
State legislation that favored cooperation and networking. 
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This chapter has presented highlights from study findings. Volume 11 will discuss 
these findings in greater detail, along with program-specific information. 

Volume II describes each program's legislative funding history, objectives, 
operations, and evaluation findings. 

A program's funding history in volume II will consist of annual authorizations and 
subsequent appropriations; program goals and objectives state the broad 
congressional intent in establishing each education program. The program 
operations sections in volume II will discuss how programs actually have been 
implemented and describe the scope of current activities. The evaluation 
findings will focus on program effectiveness and progress toward achieving long- 
range program goals. 

In subsequent years, volume II will report on program changes in detail. Volume 
I, on the other hand, will continue to cover the evaluation role of the 
Department of Education from a broader perspective. Significant changes in the 
evaluation mission will be explained along with any new analytic approaches 
adopted or established techniques that may be de~emphasized. 
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